mlsanityData to 7 Oct 2026

Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages

arXiv 2109.10534 ↗First posted 22 Sep 2021cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

We explore the impact of leveraging the relatedness of languages that belong to the same family in NLP models using multilingual fine-tuning. We hypothesize and validate that multilingual fine-tuning of pre-trained language models can yield better performance on downstream NLP applications, compared to models fine-tuned on individual languages. A first of its kind detailed study is presented to track performance change as languages are added to a base language in a graded and greedy (in the sense of best boost of performance) manner; which reveals that careful selection of subset of related languages can significantly improve performance than utilizing all related languages. The Indo-Aryan (IA) language family is chosen for the study, the exact languages being Bengali, Gujarati, Hindi, Marathi, Oriya, Punjabi and Urdu. The script barrier is crossed by simple rule-based transliteration of the text of all languages to Devanagari. Experiments are performed on mBERT, IndicBERT, MuRIL and two RoBERTa-based LMs, the last two being pre-trained by us. Low resource languages, such as Oriya and Punjabi, are found to be the largest beneficiaries of multilingual fine-tuning. Textual Entailment, Entity Classification, Section Title Prediction, tasks of IndicGLUE and POS tagging form our test bed. Compared to monolingual fine tuning we get relative performance improvement of up to 150% in the downstream tasks. The surprise take-away is that for any language there is a particular combination of other languages which yields the best performance, and any additional language is in fact detrimental.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 22 of its 43 references that are in the map. First posted on arXiv 22 Sep 2021.

Builds on

22 of 43 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  10. 2019How multilingual is Multilingual BERT?cited by 642
  11. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  12. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
10 more
  1. 2021MuRIL: Multilingual Representations for Indian Languagescited by 177
  2. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  3. 2017The IIT Bombay English-Hindi Parallel Corpuscited by 74
  4. 2019Small and Practical BERT Models for Sequence Labelingcited by 40
  5. 2019Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Frameworkcited by 26
  6. 2020Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Studycited by 24
  7. 2020Indic-Transformers: An Analysis of Transformer Language Models for Indian Languagescited by 22
  8. 2019Zero-shot Dependency Parsing with Pre-trained Multilingual Sentence Representationscited by 17
  9. 2021Analysing The Impact Of Linguistic Features On Cross-Lingual Transfercited by 16
  10. 2018Automatic Identification of Closely-related Indian Languages: Resources and Experimentscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 4; 2024: 11; 2025: 2; 2026: 4.

24 papers.

Sort:

By area:

  1. 2023Multi-lingual and Multi-cultural Figurative Language Understandingcited by 47
  2. 2024RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanizationcited by 29
  3. 2022Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languagescited by 28
  4. 2023Romanization-based Large-scale Adaptation of Multilingual Language Modelscited by 21
  5. 2021Alternative Input Signals Ease Transfer in Multilingual Machine Translationcited by 14
  6. 2022Does Transliteration Help Multilingual Language Modeling?cited by 14
  7. 2024Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuningcited by 14
  8. 2024mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humanscited by 10
  9. 2023Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embeddingcited by 9
  10. 2024TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Datacited by 9
  11. 2024Exploring the Role of Transliteration in In-Context Learning for Low-resource Languages Written in Non-Latin Scriptscited by 8
  12. 2024How Transliterations Improve Crosslingual Alignmentcited by 6
12 more
  1. 2024TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Modelscited by 3
  2. 2025Text2Cypher Across Languages: Evaluating and Finetuning LLMscited by 3
  3. 2023Multilingual Multiword Expression Identification Using Lateral Inhibition and Domain Adaptationcited by 2
  4. 2024Training a Bilingual Language Model by Mapping Tokens onto a Shared Character Spacecited by 1
  5. 2025Enhancing Cross-Lingual Transfer through Reversible Transliteration: A Huffman-Based Approach for Low-Resource Languagescited by 1
  6. 2026One Script Instead of Hundreds? On Pretraining Romanized Encoder Language Modelscited by 1
  7. 2026Making Large Language Models Speak Tulu: Structured Prompting for an Extremely Low-Resource Languagecited by 1
  8. 2024Script-Agnostic Language Identificationcited by 0
  9. 2024Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Modelcited by 0
  10. 2024IITR-CIOL@NLU of Devanagari Script Languages 2025: Multilingual Hate Speech Detection and Target Identification in Devanagari-Scripted Languagescited by 0
  11. 2026A Study of Crosslinguistic Influence in Language Modelscited by 0
  12. 2026From Sinhala to Dhivehi: Cross-Lingual Transfer Learning for Low-Resource Speech Recognitioncited by 0