mlsanityData to 14 Oct 2026

L3Cube-HindBERT and DevBERT: Pre-Trained BERT Transformer models for Devanagari based Hindi and Marathi Languages

arXiv 2211.11418 ↗First posted 21 Nov 2022cs.CL cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

The monolingual Hindi BERT models currently available on the model hub do not perform better than the multi-lingual models on downstream tasks. We present L3Cube-HindBERT, a Hindi BERT model pre-trained on Hindi monolingual corpus. Further, since Indic languages, Hindi and Marathi share the Devanagari script, we train a single model for both languages. We release DevBERT, a Devanagari BERT model trained on both Marathi and Hindi monolingual datasets. We evaluate these models on downstream Hindi and Marathi text classification and named entity recognition tasks. The HindBERT and DevBERT-based models show significant improvements over multi-lingual MuRIL, IndicBERT, and XLM-R. Based on these observations we also release monolingual BERT models for other Indic languages Kannada, Telugu, Malayalam, Tamil, Gujarati, Assamese, Odia, Bengali, and Punjabi. These models are shared at https://huggingface.co/l3cube-pune .

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 18 of its 23 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

18 of 23 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  5. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  6. 2021MuRIL: Multilingual Representations for Indian Languagescited by 179
  7. 2021BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Banglacited by 125
  8. 2022L3Cube-MahaCorpus and MahaBERT: Marathi Monolingual Corpus, Marathi BERT Language Models, and Resourcescited by 51
  9. 2022L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Modelscited by 34
  10. 2021L3CubeMahaSent: A Marathi Tweet-based Sentiment Analysis Datasetcited by 31
  11. 2022L3Cube-MahaNLP: Marathi Natural Language Processing Datasets, Models, and Librarycited by 31
  12. 2020Deep Learning for Hindi Text Classification: A Comparisoncited by 22
6 more
  1. 2020Indic-Transformers: An Analysis of Transformer Language Models for Indian Languagescited by 22
  2. 2022L3Cube-MahaHate: A Tweet-based Marathi Hate Speech Detection Dataset and BERT modelscited by 21
  3. 2022L3Cube-MahaNER: A Marathi Named Entity Recognition Dataset and BERT modelscited by 21
  4. 2021Experimental Evaluation of Deep Learning models for Marathi Text Classificationcited by 19
  5. 2022Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathicited by 16
  6. 2022Mono vs Multilingual BERT: A Case Study in Hindi and Marathi Named Entity Recognitioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 10; 2025: 8; 2026: 6.

35 papers.

Sort:

  1. 2023L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERTcited by 15
  2. 2024L3Cube-IndicNews: News-based Short Text and Long Document Classification Datasets in Indic Languagescited by 8
  3. 2024Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Modelscited by 6
  4. 2023Breaking Language Barriers: A Question Answering Dataset for Hindi and Marathicited by 5
  5. 2025IndicSQuAD: A Comprehensive Multilingual Question Answering Dataset for Indic Languagescited by 5
  6. 2023Mixed-Distil-BERT: Code-mixed Language Modeling for Bangla, English, and Hindicited by 4
  7. 2023Question answering using deep learning in low resource Indian language Marathicited by 4
  8. 2024Code-mixed Sentiment and Hate-speech Predictioncited by 4
  9. 2024Building pre-train LLM Dataset for the INDIC Languages: a case study on Hindicited by 4
  10. 2025SenWiCh: Sense-Annotation of Low-Resource Languages for WiC using Hybrid Methodscited by 4
  11. 2023When your Cousin has the Right Connections: Unsupervised Bilingual Lexicon Induction for Related Data-Imbalanced Languagescited by 3
  12. 2023L3Cube-MahaSocialNER: A Social Media based Marathi NER Dataset and BERT modelscited by 2
23 more
  1. 2024MasonTigers@LT-EDI-2024: An Ensemble Approach Towards Detecting Homophobia and Transphobia in Social Media Commentscited by 2
  2. 2024MahaSQuAD: Bridging Linguistic Divides in Marathi Question-Answeringcited by 2
  3. 2023Harnessing Pre-Trained Sentence Transformers for Offensive Language Detection in Indian Languagescited by 1
  4. 2024Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That!cited by 1
  5. 2024MasonTigers at SemEval-2024 Task 1: An Ensemble Approach for Semantic Textual Relatednesscited by 1
  6. 2025Unmask It! AI-Generated Product Review Detection in Dravidian Languagescited by 1
  7. 2025L3Cube-IndicHeadline-ID: A Dataset for Headline Identification and Semantic Evaluation in Low-Resource Indian Languagescited by 1
  8. 2025MaiBERT: A Pre-training Corpus and Language Model for Low-Resourced Maithili Languagecited by 1
  9. 2025Tourism Question Answer System in Indian Language using Domain-Adapted Foundation Modelscited by 1
  10. 2023Leveraging Language Identification to Enhance Code-Mixed Text Classificationcited by 0
  11. 2023Mavericks at BLP-2023 Task 1: Ensemble-based Approach Using Language Models for Violence Inciting Text Detectioncited by 0
  12. 2023On Significance of Subword tokenization for Low Resource and Efficient Named Entity Recognition: A case study in Marathicited by 0
  13. 2023HCDIR: End-to-end Hate Context Detection, and Intensity Reduction model for online commentscited by 0
  14. 2024Multilingual Evaluation of Semantic Textual Relatednesscited by 0
  15. 2024IITR-CIOL@NLU of Devanagari Script Languages 2025: Multilingual Hate Speech Detection and Target Identification in Devanagari-Scripted Languagescited by 0
  16. 2025Human-Centered Supervision for Sentiment Analysis in Telugu: A Systematic Inquiry Beyond Accuracycited by 0
  17. 2025Multilingual Datasets for Custom Input Extraction and Explanation Requests Parsing in Conversational XAI Systemscited by 0
  18. 2026One Instruction Does Not Fit All: How Well Do Embeddings Align Personas and Instructions in Low-Resource Indian Languages?cited by 0
  19. 2026BhashaSetu: Cross-Lingual Knowledge Transfer from High-Resource to Extreme Low-Resource Languagescited by 0
  20. 2026Benchmarking BERT-based Models for Sentence-level Topic Classification in Nepali Languagecited by 0
  21. 2026Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindicited by 0
  22. 2026MKJ at SemEval-2026 Task 9: A Comparative Study of Generalist, Specialist, and Ensemble Strategies for Multilingual Polarizationcited by 0
  23. 2026HomoEnsNER: Does Language Alignment Outperform Architectural Complexity in Gujarati Named Entity Recognition?cited by 0