# L3Cube-MahaSBERT and HindSBERT: Sentence BERT Models and Benchmarking BERT Sentence Representations for Hindi and Marathi arXiv: https://arxiv.org/abs/2211.11187 ยท HTML: https://mlsanity.com/paper/2211.11187/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 32 references that are in the map. - First posted on arXiv 21 Nov 2022. ## Builds on (most cited first, 28 of 28) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020) - [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018) - [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018) - [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017) - [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018) - [MuRIL: Multilingual Representations for Indian Languages](https://mlsanity.com/paper/2103.10730/) (2021) - [On the Sentence Embeddings from Pre-trained Language Models](https://mlsanity.com/paper/2011.05864/) (2020) - [SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models](https://mlsanity.com/paper/2002.06652/) (2020) - [L3Cube-MahaCorpus and MahaBERT: Marathi Monolingual Corpus, Marathi BERT Language Models, and Resources](https://mlsanity.com/paper/2202.01159/) (2022) - [GottBERT: a pure German Language Model](https://mlsanity.com/paper/2012.02110/) (2020) - [Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages](https://mlsanity.com/paper/2205.12654/) (2022) - [AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages](https://mlsanity.com/paper/2005.00085/) (2020) - [Evaluating Compositionality in Sentence Embeddings](https://mlsanity.com/paper/1802.04302/) (2018) - [RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model](https://mlsanity.com/paper/2105.11314/) (2021) - [Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream NLP Tasks](https://mlsanity.com/paper/2101.10642/) (2021) - [L3CubeMahaSent: A Marathi Tweet-based Sentiment Analysis Dataset](https://mlsanity.com/paper/2103.11408/) (2021) - [L3Cube-MahaNLP: Marathi Natural Language Processing Datasets, Models, and Library](https://mlsanity.com/paper/2205.14728/) (2022) - [Evaluation of sentence embeddings in downstream and linguistic probing tasks](https://mlsanity.com/paper/1806.06259/) (2018) - [Universal Text Representation from BERT: An Empirical Study](https://mlsanity.com/paper/1910.07973/) (2019) - [IndicXNLI: Evaluating Multilingual Inference for Indian Languages](https://mlsanity.com/paper/2204.08776/) (2022) - [Deep Learning for Hindi Text Classification: A Comparison](https://mlsanity.com/paper/2001.10340/) (2020) - [Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathi](https://mlsanity.com/paper/2204.08669/) (2022) - [Sentence Similarity Based on Contexts](https://mlsanity.com/paper/2105.07623/) (2021) - [Don't Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Pooling](https://mlsanity.com/paper/2209.05972/) (2022) - [Spread Love Not Hate: Undermining the Importance of Hateful Pre-training for Hate Speech Detection](https://mlsanity.com/paper/2210.04267/) (2022) ## Built on by (most cited first, 10 of 10) - [L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT](https://mlsanity.com/paper/2304.11434/) (2023) - [L3Cube-IndicNews: News-based Short Text and Long Document Classification Datasets in Indic Languages](https://mlsanity.com/paper/2401.02254/) (2024) - [mahaNLP: A Marathi Natural Language Processing Library](https://mlsanity.com/paper/2311.02579/) (2023) - [BERTopic for Topic Modeling of Hindi Short Texts: A Comparative Study](https://mlsanity.com/paper/2501.03843/) (2025) - [Harnessing Pre-Trained Sentence Transformers for Offensive Language Detection in Indian Languages](https://mlsanity.com/paper/2310.02249/) (2023) - [Towards Building Efficient Sentence BERT Models using Layer Pruning](https://mlsanity.com/paper/2409.14168/) (2024) - [Enhancing Plagiarism Detection in Marathi with a Weighted Ensemble of TF-IDF and BERT Embeddings for Low-Resource Language Processing](https://mlsanity.com/paper/2501.05260/) (2025) - [MahaParaphrase: A Marathi Paraphrase Detection Corpus and BERT-based Models](https://mlsanity.com/paper/2508.17444/) (2025) - [L3Cube-MahaSTS: A Marathi Sentence Similarity Dataset and Models](https://mlsanity.com/paper/2508.21569/) (2025) - [Multilingual Evaluation of Semantic Textual Relatedness](https://mlsanity.com/paper/2404.09047/) (2024) ## Abstract (arXiv) Sentence representation from vanilla BERT models does not work well on sentence similarity tasks. Sentence-BERT models specifically trained on STS or NLI datasets are shown to provide state-of-the-art performance. However, building these models for low-resource languages is not straightforward due to the lack of these specialized datasets. This work focuses on two low-resource Indian languages, Hindi and Marathi. We train sentence-BERT models for these languages using synthetic NLI and STS datasets prepared using machine translation. We show that the strategy of NLI pre-training followed by STSb fine-tuning is effective in generating high-performance sentence-similarity models for Hindi and Marathi. The vanilla BERT models trained using this simple strategy outperform the multilingual LaBSE trained using a complex training strategy. These models are evaluated on downstream text classification and similarity tasks. We evaluate these models on real text classification datasets to show embeddings obtained from synthetic data training are generalizable to real datasets as well and thus represent an effective training strategy for low-resource languages. We also provide a comparative analysis of sentence embeddings from fast text models, multilingual BERT models (mBERT, IndicBERT, xlm-RoBERTa, MuRIL), multilingual sentence embedding models (LASER, LaBSE), and monolingual BERT models based on L3Cube-MahaBERT and HindBERT. We release L3Cube-MahaSBERT and HindSBERT, the state-of-the-art sentence-BERT models for Marathi and Hindi respectively. Our work also serves as a guide to building low-resource sentence embedding models.