# Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Study

arXiv: https://arxiv.org/abs/2106.03958 · HTML: https://mlsanity.com/paper/2106.03958/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [MuRIL: Multilingual Representations for Indian Languages](https://mlsanity.com/paper/2103.10730/) (2021)
- [Multilingual Alignment of Contextual Word Representations](https://mlsanity.com/paper/2002.03518/) (2020)
- [UNKs Everywhere: Adapting Multilingual Language Models to New Scripts](https://mlsanity.com/paper/2012.15562/) (2020)
- [Extending Multilingual BERT to Low-Resource Languages](https://mlsanity.com/paper/2004.13640/) (2020)
- [Rapid Adaptation of Neural Machine Translation to New Languages](https://mlsanity.com/paper/1808.04189/) (2018)
- [UDapter: Language Adaptation for Truly Universal Dependency Parsing](https://mlsanity.com/paper/2004.14327/) (2020)
- [Generalized Data Augmentation for Low-Resource Translation](https://mlsanity.com/paper/1906.03785/) (2019)
- [Multilingual Neural Machine Translation With Soft Decoupled Encoding](https://mlsanity.com/paper/1902.03499/) (2019)
- [Unsupervised Cross-lingual Transfer of Word Embedding Spaces](https://mlsanity.com/paper/1809.03633/) (2018)
- [Bilingual Lexicon Induction through Unsupervised Machine Translation](https://mlsanity.com/paper/1907.10761/) (2019)
- [Utilizing Language Relatedness to improve Machine Translation: A Case Study on Languages of the Indian Subcontinent](https://mlsanity.com/paper/2003.08925/) (2020)
- [Improving Target-side Lexical Transfer in Multilingual Neural Machine Translation](https://mlsanity.com/paper/2010.01667/) (2020)
- [Embedding time expressions for deep temporal ordering models](https://mlsanity.com/paper/1906.08287/) (2019)

## Built on by (most cited first, 19 of 19)

- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages](https://mlsanity.com/paper/2212.05409/) (2022)
- [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022)
- [Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation](https://mlsanity.com/paper/2203.09435/) (2022)
- [When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer](https://mlsanity.com/paper/2110.14782/) (2021)
- [IndicBART: A Pre-trained Model for Indic Natural Language Generation](https://mlsanity.com/paper/2109.02903/) (2021)
- [RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanization](https://mlsanity.com/paper/2401.14280/) (2024)
- [Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages](https://mlsanity.com/paper/2203.01976/) (2022)
- [Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users](https://mlsanity.com/paper/2205.03018/) (2022)
- [Does Transliteration Help Multilingual Language Modeling?](https://mlsanity.com/paper/2201.12501/) (2022)
- [Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment](https://mlsanity.com/paper/2406.19759/) (2024)
- [ALIGN-MLM: Word Embedding Alignment is Crucial for Multilingual Pre-training](https://mlsanity.com/paper/2211.08547/) (2022)
- [Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challenges](https://mlsanity.com/paper/2509.11570/) (2025)
- [Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Models](https://mlsanity.com/paper/2210.15184/) (2022)
- [Hate Speech Detection in Limited Data Contexts using Synthetic Data Generation](https://mlsanity.com/paper/2310.02876/) (2023)
- [CharSpan: Utilizing Lexical Similarity to Enable Zero-Shot Machine Translation for Extremely Low-resource Languages](https://mlsanity.com/paper/2305.05214/) (2023)
- [NepTam: A Nepali-Tamang Parallel Corpus and Baseline Machine Translation Experiments](https://mlsanity.com/paper/2603.14053/) (2026)
- [The Latin Substrate: How Language Models Represent and Mediate Script Choice](https://mlsanity.com/paper/2605.31363/) (2026)
- [Rethinking Indic AI from a Lens of Cultural Heritage Preservation](https://mlsanity.com/paper/2607.06544/) (2026)

## Abstract (arXiv)

Recent research in multilingual language models (LM) has demonstrated their ability to effectively handle multiple languages in a single model. This holds promise for low web-resource languages (LRL) as multilingual models can enable transfer of supervision from high resource languages to LRLs. However, incorporating a new language in an LM still remains a challenge, particularly for languages with limited corpora and in unseen scripts. In this paper we argue that relatedness among languages in a language family may be exploited to overcome some of the corpora limitations of LRLs, and propose RelateLM. We focus on Indian languages, and exploit relatedness along two dimensions: (1) script (since many Indic scripts originated from the Brahmic script), and (2) sentence structure. RelateLM uses transliteration to convert the unseen script of limited LRL text into the script of a Related Prominent Language (RPL) (Hindi in our case). While exploiting similar sentence structures, RelateLM utilizes readily available bilingual dictionaries to pseudo translate RPL text into LRL corpora. Experiments on multiple real-world benchmark datasets provide validation to our hypothesis that using a related language as pivot, along with transliteration and pseudo translation based data augmentation, can be an effective way to adapt LMs for LRLs, rather than direct training or pivoting through English.