# L3Cube-MahaCorpus and MahaBERT: Marathi Monolingual Corpus, Marathi BERT Language Models, and Resources

arXiv: https://arxiv.org/abs/2202.01159 · HTML: https://mlsanity.com/paper/2202.01159/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 2 Feb 2022.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](https://mlsanity.com/paper/2101.01785/) (2020)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://mlsanity.com/paper/1912.05372/) (2019)
- [PhoBERT: Pre-trained language models for Vietnamese](https://mlsanity.com/paper/2003.00744/) (2020)
- [RobBERT: a Dutch RoBERTa-based Language Model](https://mlsanity.com/paper/2001.06286/) (2020)
- [GottBERT: a pure German Language Model](https://mlsanity.com/paper/2012.02110/) (2020)
- [RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model](https://mlsanity.com/paper/2105.11314/) (2021)
- [L3CubeMahaSent: A Marathi Tweet-based Sentiment Analysis Dataset](https://mlsanity.com/paper/2103.11408/) (2021)
- [Cross-lingual Offensive Language Identification for Low Resource Languages: The Case of Marathi](https://mlsanity.com/paper/2109.03552/) (2021)
- [iNLTK: Natural Language Toolkit for Indic Languages](https://mlsanity.com/paper/2009.12534/) (2020)
- [Deep Learning for Hindi Text Classification: A Comparison](https://mlsanity.com/paper/2001.10340/) (2020)
- [Indic-Transformers: An Analysis of Transformer Language Models for Indian Languages](https://mlsanity.com/paper/2011.02323/) (2020)
- [Experimental Evaluation of Deep Learning models for Marathi Text Classification](https://mlsanity.com/paper/2101.04899/) (2021)
- [Hate and Offensive Speech Detection in Hindi and Marathi](https://mlsanity.com/paper/2110.12200/) (2021)
- [Overview of the HASOC Subtrack at FIRE 2021: Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages](https://mlsanity.com/paper/2112.09301/) (2021)
- [Evaluation of Deep Learning Models for Hostility Detection in Hindi Text](https://mlsanity.com/paper/2101.04144/) (2021)
- [Contextual Hate Speech Detection in Code Mixed Text using Transformer Based Approaches](https://mlsanity.com/paper/2110.09338/) (2021)

## Built on by (most cited first, 30 of 51)

- [L3Cube-HindBERT and DevBERT: Pre-Trained BERT Transformer models for Devanagari based Hindi and Marathi Languages](https://mlsanity.com/paper/2211.11418/) (2022)
- [L3Cube-MahaNLP: Marathi Natural Language Processing Datasets, Models, and Library](https://mlsanity.com/paper/2205.14728/) (2022)
- [L3Cube-MahaHate: A Tweet-based Marathi Hate Speech Detection Dataset and BERT models](https://mlsanity.com/paper/2203.13778/) (2022)
- [L3Cube-MahaNER: A Marathi Named Entity Recognition Dataset and BERT models](https://mlsanity.com/paper/2204.06029/) (2022)
- [Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathi](https://mlsanity.com/paper/2204.08669/) (2022)
- [L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT](https://mlsanity.com/paper/2304.11434/) (2023)
- [L3Cube-MahaSBERT and HindSBERT: Sentence BERT Models and Benchmarking BERT Sentence Representations for Hindi and Marathi](https://mlsanity.com/paper/2211.11187/) (2022)
- [L3Cube-IndicNews: News-based Short Text and Long Document Classification Datasets in Indic Languages](https://mlsanity.com/paper/2401.02254/) (2024)
- [L3Cube-MahaSent-MD: A Multi-domain Marathi Sentiment Analysis Dataset and Transformer Models](https://mlsanity.com/paper/2306.13888/) (2023)
- [L3Cube-MahaNews: News-based Short Text and Long Document Classification Datasets in Marathi](https://mlsanity.com/paper/2404.18216/) (2024)
- [Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challenges](https://mlsanity.com/paper/2509.11570/) (2025)
- [Mono vs Multilingual BERT: A Case Study in Hindi and Marathi Named Entity Recognition](https://mlsanity.com/paper/2203.12907/) (2022)
- [Overview of the HASOC Subtrack at FIRE 2022: Offensive Language Identification in Marathi](https://mlsanity.com/paper/2211.10163/) (2022)
- [Breaking Language Barriers: A Question Answering Dataset for Hindi and Marathi](https://mlsanity.com/paper/2308.09862/) (2023)
- [Chain-of-Translation Prompting (CoTR): A Novel Prompting Technique for Low Resource Languages](https://mlsanity.com/paper/2409.04512/) (2024)
- [IndicSQuAD: A Comprehensive Multilingual Question Answering Dataset for Indic Languages](https://mlsanity.com/paper/2505.03688/) (2025)
- [Spread Love Not Hate: Undermining the Importance of Hateful Pre-training for Hate Speech Detection](https://mlsanity.com/paper/2210.04267/) (2022)
- [Question answering using deep learning in low resource Indian language Marathi](https://mlsanity.com/paper/2309.15779/) (2023)
- [A Twitter BERT Approach for Offensive Language Detection in Marathi](https://mlsanity.com/paper/2212.10039/) (2022)
- [mahaNLP: A Marathi Natural Language Processing Library](https://mlsanity.com/paper/2311.02579/) (2023)
- [My Boli: Code-mixed Marathi-English Corpora, Pretrained Language Models and Evaluation Benchmarks](https://mlsanity.com/paper/2306.14030/) (2023)
- [Robust Sentiment Analysis for Low Resource languages Using Data Augmentation Approaches: A Case Study in Marathi](https://mlsanity.com/paper/2310.00734/) (2023)
- [L3Cube-MahaSocialNER: A Social Media based Marathi NER Dataset and BERT models](https://mlsanity.com/paper/2401.00170/) (2023)
- [MasonTigers@LT-EDI-2024: An Ensemble Approach Towards Detecting Homophobia and Transphobia in Social Media Comments](https://mlsanity.com/paper/2401.14681/) (2024)
- [MahaSQuAD: Bridging Linguistic Divides in Marathi Question-Answering](https://mlsanity.com/paper/2404.13364/) (2024)
- [L3Cube-MahaEmotions: A Marathi Emotion Recognition Dataset with Synthetic Annotations using CoTR prompting and Large Language Models](https://mlsanity.com/paper/2506.00863/) (2025)
- [Towards Simple and Efficient Task-Adaptive Pre-training for Text Classification](https://mlsanity.com/paper/2209.12943/) (2022)
- [MasonTigers at SemEval-2024 Task 1: An Ensemble Approach for Semantic Textual Relatedness](https://mlsanity.com/paper/2403.14990/) (2024)
- [L3Cube-MahaSum: A Comprehensive Dataset and BART Models for Abstractive Text Summarization in Marathi](https://mlsanity.com/paper/2410.09184/) (2024)
- [Long Range Named Entity Recognition for Marathi Documents](https://mlsanity.com/paper/2410.09192/) (2024)

## Abstract (arXiv)

We present L3Cube-MahaCorpus a Marathi monolingual data set scraped from different internet sources. We expand the existing Marathi monolingual corpus with 24.8M sentences and 289M tokens. We further present, MahaBERT, MahaAlBERT, and MahaRoBerta all BERT-based masked language models, and MahaFT, the fast text word embeddings both trained on full Marathi corpus with 752M tokens. We show the effectiveness of these resources on downstream Marathi sentiment analysis, text classification, and named entity recognition (NER) tasks. We also release MahaGPT, a generative Marathi GPT model trained on Marathi corpus. Marathi is a popular language in India but still lacks these resources. This work is a step forward in building open resources for the Marathi language. The data and models are available at https://github.com/l3cube-pune/MarathiNLP .