# Indic-Transformers: An Analysis of Transformer Language Models for Indian Languages

arXiv: https://arxiv.org/abs/2011.02323 · HTML: https://mlsanity.com/paper/2011.02323/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2020.

## Built on by (most cited first, 22 of 22)

- [Distributed Deep Learning in Open Collaborations](https://mlsanity.com/paper/2106.10207/) (2021)
- [L3Cube-MahaCorpus and MahaBERT: Marathi Monolingual Corpus, Marathi BERT Language Models, and Resources](https://mlsanity.com/paper/2202.01159/) (2022)
- [L3Cube-HindBERT and DevBERT: Pre-Trained BERT Transformer models for Devanagari based Hindi and Marathi Languages](https://mlsanity.com/paper/2211.11418/) (2022)
- [IndicXNLI: Evaluating Multilingual Inference for Indian Languages](https://mlsanity.com/paper/2204.08776/) (2022)
- [Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages](https://mlsanity.com/paper/2109.10534/) (2021)
- [Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathi](https://mlsanity.com/paper/2204.08669/) (2022)
- [A Review of Bangla Natural Language Processing Tasks and the Utility of Transformer Models](https://mlsanity.com/paper/2107.03844/) (2021)
- [DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culture](https://mlsanity.com/paper/2509.19274/) (2025)
- [Computational historical linguistics and language diversity in South Asia](https://mlsanity.com/paper/2203.12524/) (2022)
- ['Since Lawyers are Males..': Examining Implicit Gender Bias in Hindi Language Generation by LLMs](https://mlsanity.com/paper/2409.13484/) (2024)
- [Hope Speech detection in under-resourced Kannada language](https://mlsanity.com/paper/2108.04616/) (2021)
- [Hostility Detection in Hindi leveraging Pre-Trained Language Models](https://mlsanity.com/paper/2101.05494/) (2021)
- [L3Cube-MahaNews: News-based Short Text and Long Document Classification Datasets in Marathi](https://mlsanity.com/paper/2404.18216/) (2024)
- [Decoding the Diversity: A Review of the Indic AI Research Landscape](https://mlsanity.com/paper/2406.09559/) (2024)
- [Vy=akarana: A Colorless Green Benchmark for Syntactic Evaluation in Indic Languages](https://mlsanity.com/paper/2103.00854/) (2021)
- [Milestones in Bengali Sentiment Analysis leveraging Transformer-models: Fundamentals, Challenges and Future Directions](https://mlsanity.com/paper/2401.07847/) (2024)
- [On Importance of Pruning and Distillation for Efficient Low Resource NLP](https://mlsanity.com/paper/2409.14162/) (2024)
- [Improving the Efficiency of Long Document Classification using Sentence Ranking Approach](https://mlsanity.com/paper/2506.07248/) (2025)
- [HinFlair: pre-trained contextual string embeddings for pos tagging and text classification in the Hindi language](https://mlsanity.com/paper/2101.06949/) (2021)
- [MASALA: Modelling and Analysing the Semantics of Adpositions in Linguistic Annotation of Hindi](https://mlsanity.com/paper/2205.03955/) (2022)
- [Efficient Zero-Shot Long Document Classification by Reducing Context Through Sentence Ranking](https://mlsanity.com/paper/2508.17490/) (2025)
- [Multilingual Conversational AI for Financial Assistance: Bridging Language Barriers in Indian FinTech](https://mlsanity.com/paper/2512.01439/) (2025)

## Abstract (arXiv)

Language models based on the Transformer architecture have achieved state-of-the-art performance on a wide range of NLP tasks such as text classification, question-answering, and token classification. However, this performance is usually tested and reported on high-resource languages, like English, French, Spanish, and German. Indian languages, on the other hand, are underrepresented in such benchmarks. Despite some Indian languages being included in training multilingual Transformer models, they have not been the primary focus of such work. In order to evaluate the performance on Indian languages specifically, we analyze these language models through extensive experiments on multiple downstream tasks in Hindi, Bengali, and Telugu language. Here, we compare the efficacy of fine-tuning model parameters of pre-trained models against that of training a language model from scratch. Moreover, we empirically argue against the strict dependency between the dataset size and model performance, but rather encourage task-specific model and method selection. We achieve state-of-the-art performance on Hindi and Bengali languages for text classification task. Finally, we present effective strategies for handling the modeling of Indian languages and we release our model checkpoints for the community : https://huggingface.co/neuralspace-reverie.