# Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages

arXiv: https://arxiv.org/abs/2109.10534 · HTML: https://mlsanity.com/paper/2109.10534/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 43 references that are in the map.
- First posted on arXiv 22 Sep 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [MuRIL: Multilingual Representations for Indian Languages](https://mlsanity.com/paper/2103.10730/) (2021)
- [Neural Machine Translation with Byte-Level Subwords](https://mlsanity.com/paper/1909.03341/) (2019)
- [The IIT Bombay English-Hindi Parallel Corpus](https://mlsanity.com/paper/1710.02855/) (2017)
- [Small and Practical BERT Models for Sequence Labeling](https://mlsanity.com/paper/1909.00100/) (2019)
- [Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework](https://mlsanity.com/paper/1910.04708/) (2019)
- [Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study](https://mlsanity.com/paper/2009.14304/) (2020)
- [Indic-Transformers: An Analysis of Transformer Language Models for Indian Languages](https://mlsanity.com/paper/2011.02323/) (2020)
- [Zero-shot Dependency Parsing with Pre-trained Multilingual Sentence Representations](https://mlsanity.com/paper/1910.05479/) (2019)
- [Analysing The Impact Of Linguistic Features On Cross-Lingual Transfer](https://mlsanity.com/paper/2105.05975/) (2021)
- [Automatic Identification of Closely-related Indian Languages: Resources and Experiments](https://mlsanity.com/paper/1803.09405/) (2018)

## Built on by (most cited first, 24 of 24)

- [Multi-lingual and Multi-cultural Figurative Language Understanding](https://mlsanity.com/paper/2305.16171/) (2023)
- [RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanization](https://mlsanity.com/paper/2401.14280/) (2024)
- [Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languages](https://mlsanity.com/paper/2212.10168/) (2022)
- [Romanization-based Large-scale Adaptation of Multilingual Language Models](https://mlsanity.com/paper/2304.08865/) (2023)
- [Alternative Input Signals Ease Transfer in Multilingual Machine Translation](https://mlsanity.com/paper/2110.07804/) (2021)
- [Does Transliteration Help Multilingual Language Modeling?](https://mlsanity.com/paper/2201.12501/) (2022)
- [Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning](https://mlsanity.com/paper/2411.18571/) (2024)
- [mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humans](https://mlsanity.com/paper/2406.04215/) (2024)
- [Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embedding](https://mlsanity.com/paper/2309.08929/) (2023)
- [TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data](https://mlsanity.com/paper/2405.09913/) (2024)
- [Exploring the Role of Transliteration in In-Context Learning for Low-resource Languages Written in Non-Latin Scripts](https://mlsanity.com/paper/2407.02320/) (2024)
- [How Transliterations Improve Crosslingual Alignment](https://mlsanity.com/paper/2409.17326/) (2024)
- [TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2401.06620/) (2024)
- [Text2Cypher Across Languages: Evaluating and Finetuning LLMs](https://mlsanity.com/paper/2506.21445/) (2025)
- [Multilingual Multiword Expression Identification Using Lateral Inhibition and Domain Adaptation](https://mlsanity.com/paper/2306.10419/) (2023)
- [Training a Bilingual Language Model by Mapping Tokens onto a Shared Character Space](https://mlsanity.com/paper/2402.16065/) (2024)
- [Enhancing Cross-Lingual Transfer through Reversible Transliteration: A Huffman-Based Approach for Low-Resource Languages](https://mlsanity.com/paper/2509.17493/) (2025)
- [One Script Instead of Hundreds? On Pretraining Romanized Encoder Language Models](https://mlsanity.com/paper/2601.05776/) (2026)
- [Making Large Language Models Speak Tulu: Structured Prompting for an Extremely Low-Resource Language](https://mlsanity.com/paper/2602.15378/) (2026)
- [Script-Agnostic Language Identification](https://mlsanity.com/paper/2406.17901/) (2024)
- [Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Model](https://mlsanity.com/paper/2410.16006/) (2024)
- [IITR-CIOL@NLU of Devanagari Script Languages 2025: Multilingual Hate Speech Detection and Target Identification in Devanagari-Scripted Languages](https://mlsanity.com/paper/2412.17947/) (2024)
- [A Study of Crosslinguistic Influence in Language Models](https://mlsanity.com/paper/2601.21587/) (2026)
- [From Sinhala to Dhivehi: Cross-Lingual Transfer Learning for Low-Resource Speech Recognition](https://mlsanity.com/paper/2607.06289/) (2026)

## Abstract (arXiv)

We explore the impact of leveraging the relatedness of languages that belong to the same family in NLP models using multilingual fine-tuning. We hypothesize and validate that multilingual fine-tuning of pre-trained language models can yield better performance on downstream NLP applications, compared to models fine-tuned on individual languages. A first of its kind detailed study is presented to track performance change as languages are added to a base language in a graded and greedy (in the sense of best boost of performance) manner; which reveals that careful selection of subset of related languages can significantly improve performance than utilizing all related languages. The Indo-Aryan (IA) language family is chosen for the study, the exact languages being Bengali, Gujarati, Hindi, Marathi, Oriya, Punjabi and Urdu. The script barrier is crossed by simple rule-based transliteration of the text of all languages to Devanagari. Experiments are performed on mBERT, IndicBERT, MuRIL and two RoBERTa-based LMs, the last two being pre-trained by us. Low resource languages, such as Oriya and Punjabi, are found to be the largest beneficiaries of multilingual fine-tuning. Textual Entailment, Entity Classification, Section Title Prediction, tasks of IndicGLUE and POS tagging form our test bed. Compared to monolingual fine tuning we get relative performance improvement of up to 150% in the downstream tasks. The surprise take-away is that for any language there is a particular combination of other languages which yields the best performance, and any additional language is in fact detrimental.