# L3Cube-HindBERT and DevBERT: Pre-Trained BERT Transformer models for Devanagari based Hindi and Marathi Languages

arXiv: https://arxiv.org/abs/2211.11418 · HTML: https://mlsanity.com/paper/2211.11418/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 23 references that are in the map.
- First posted on arXiv 21 Nov 2022.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [MuRIL: Multilingual Representations for Indian Languages](https://mlsanity.com/paper/2103.10730/) (2021)
- [BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla](https://mlsanity.com/paper/2101.00204/) (2021)
- [L3Cube-MahaCorpus and MahaBERT: Marathi Monolingual Corpus, Marathi BERT Language Models, and Resources](https://mlsanity.com/paper/2202.01159/) (2022)
- [L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models](https://mlsanity.com/paper/2204.08398/) (2022)
- [L3CubeMahaSent: A Marathi Tweet-based Sentiment Analysis Dataset](https://mlsanity.com/paper/2103.11408/) (2021)
- [L3Cube-MahaNLP: Marathi Natural Language Processing Datasets, Models, and Library](https://mlsanity.com/paper/2205.14728/) (2022)
- [Deep Learning for Hindi Text Classification: A Comparison](https://mlsanity.com/paper/2001.10340/) (2020)
- [Indic-Transformers: An Analysis of Transformer Language Models for Indian Languages](https://mlsanity.com/paper/2011.02323/) (2020)
- [L3Cube-MahaHate: A Tweet-based Marathi Hate Speech Detection Dataset and BERT models](https://mlsanity.com/paper/2203.13778/) (2022)
- [L3Cube-MahaNER: A Marathi Named Entity Recognition Dataset and BERT models](https://mlsanity.com/paper/2204.06029/) (2022)
- [Experimental Evaluation of Deep Learning models for Marathi Text Classification](https://mlsanity.com/paper/2101.04899/) (2021)
- [Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathi](https://mlsanity.com/paper/2204.08669/) (2022)
- [Mono vs Multilingual BERT: A Case Study in Hindi and Marathi Named Entity Recognition](https://mlsanity.com/paper/2203.12907/) (2022)

## Built on by (most cited first, 30 of 35)

- [L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT](https://mlsanity.com/paper/2304.11434/) (2023)
- [L3Cube-IndicNews: News-based Short Text and Long Document Classification Datasets in Indic Languages](https://mlsanity.com/paper/2401.02254/) (2024)
- [Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Models](https://mlsanity.com/paper/2403.12809/) (2024)
- [Breaking Language Barriers: A Question Answering Dataset for Hindi and Marathi](https://mlsanity.com/paper/2308.09862/) (2023)
- [IndicSQuAD: A Comprehensive Multilingual Question Answering Dataset for Indic Languages](https://mlsanity.com/paper/2505.03688/) (2025)
- [Mixed-Distil-BERT: Code-mixed Language Modeling for Bangla, English, and Hindi](https://mlsanity.com/paper/2309.10272/) (2023)
- [Question answering using deep learning in low resource Indian language Marathi](https://mlsanity.com/paper/2309.15779/) (2023)
- [Code-mixed Sentiment and Hate-speech Prediction](https://mlsanity.com/paper/2405.12929/) (2024)
- [Building pre-train LLM Dataset for the INDIC Languages: a case study on Hindi](https://mlsanity.com/paper/2407.09855/) (2024)
- [SenWiCh: Sense-Annotation of Low-Resource Languages for WiC using Hybrid Methods](https://mlsanity.com/paper/2505.23714/) (2025)
- [When your Cousin has the Right Connections: Unsupervised Bilingual Lexicon Induction for Related Data-Imbalanced Languages](https://mlsanity.com/paper/2305.14012/) (2023)
- [L3Cube-MahaSocialNER: A Social Media based Marathi NER Dataset and BERT models](https://mlsanity.com/paper/2401.00170/) (2023)
- [MasonTigers@LT-EDI-2024: An Ensemble Approach Towards Detecting Homophobia and Transphobia in Social Media Comments](https://mlsanity.com/paper/2401.14681/) (2024)
- [MahaSQuAD: Bridging Linguistic Divides in Marathi Question-Answering](https://mlsanity.com/paper/2404.13364/) (2024)
- [Harnessing Pre-Trained Sentence Transformers for Offensive Language Detection in Indian Languages](https://mlsanity.com/paper/2310.02249/) (2023)
- [Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That!](https://mlsanity.com/paper/2403.10963/) (2024)
- [MasonTigers at SemEval-2024 Task 1: An Ensemble Approach for Semantic Textual Relatedness](https://mlsanity.com/paper/2403.14990/) (2024)
- [Unmask It! AI-Generated Product Review Detection in Dravidian Languages](https://mlsanity.com/paper/2503.09289/) (2025)
- [L3Cube-IndicHeadline-ID: A Dataset for Headline Identification and Semantic Evaluation in Low-Resource Indian Languages](https://mlsanity.com/paper/2509.02503/) (2025)
- [MaiBERT: A Pre-training Corpus and Language Model for Low-Resourced Maithili Language](https://mlsanity.com/paper/2509.15048/) (2025)
- [Tourism Question Answer System in Indian Language using Domain-Adapted Foundation Models](https://mlsanity.com/paper/2511.23235/) (2025)
- [Leveraging Language Identification to Enhance Code-Mixed Text Classification](https://mlsanity.com/paper/2306.04964/) (2023)
- [Mavericks at BLP-2023 Task 1: Ensemble-based Approach Using Language Models for Violence Inciting Text Detection](https://mlsanity.com/paper/2311.18778/) (2023)
- [On Significance of Subword tokenization for Low Resource and Efficient Named Entity Recognition: A case study in Marathi](https://mlsanity.com/paper/2312.01306/) (2023)
- [HCDIR: End-to-end Hate Context Detection, and Intensity Reduction model for online comments](https://mlsanity.com/paper/2312.13193/) (2023)
- [Multilingual Evaluation of Semantic Textual Relatedness](https://mlsanity.com/paper/2404.09047/) (2024)
- [IITR-CIOL@NLU of Devanagari Script Languages 2025: Multilingual Hate Speech Detection and Target Identification in Devanagari-Scripted Languages](https://mlsanity.com/paper/2412.17947/) (2024)
- [Human-Centered Supervision for Sentiment Analysis in Telugu: A Systematic Inquiry Beyond Accuracy](https://mlsanity.com/paper/2508.01486/) (2025)
- [Multilingual Datasets for Custom Input Extraction and Explanation Requests Parsing in Conversational XAI Systems](https://mlsanity.com/paper/2508.14982/) (2025)
- [One Instruction Does Not Fit All: How Well Do Embeddings Align Personas and Instructions in Low-Resource Indian Languages?](https://mlsanity.com/paper/2601.10205/) (2026)

## Abstract (arXiv)

The monolingual Hindi BERT models currently available on the model hub do not perform better than the multi-lingual models on downstream tasks. We present L3Cube-HindBERT, a Hindi BERT model pre-trained on Hindi monolingual corpus. Further, since Indic languages, Hindi and Marathi share the Devanagari script, we train a single model for both languages. We release DevBERT, a Devanagari BERT model trained on both Marathi and Hindi monolingual datasets. We evaluate these models on downstream Hindi and Marathi text classification and named entity recognition tasks. The HindBERT and DevBERT-based models show significant improvements over multi-lingual MuRIL, IndicBERT, and XLM-R. Based on these observations we also release monolingual BERT models for other Indic languages Kannada, Telugu, Malayalam, Tamil, Gujarati, Assamese, Odia, Bengali, and Punjabi. These models are shared at https://huggingface.co/l3cube-pune .