# Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroese

arXiv: https://arxiv.org/abs/2304.08823 · HTML: https://mlsanity.com/paper/2304.08823/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 18 Apr 2023.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://mlsanity.com/paper/2205.06266/) (2022)
- [UNKs Everywhere: Adapting Multilingual Language Models to New Scripts](https://mlsanity.com/paper/2012.15562/) (2020)
- [WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models](https://mlsanity.com/paper/2112.06598/) (2021)
- [On the Limitations of Unsupervised Bilingual Dictionary Induction](https://mlsanity.com/paper/1805.03620/) (2018)
- [A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters](https://mlsanity.com/paper/2012.15682/) (2020)
- [Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation](https://mlsanity.com/paper/2203.09435/) (2022)
- [Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spaces](https://mlsanity.com/paper/1908.06625/) (2019)
- [Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability](https://mlsanity.com/paper/2203.10753/) (2022)
- [Phylogeny-Inspired Adaptation of Multilingual Models to New Languages](https://mlsanity.com/paper/2205.09634/) (2022)
- [Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer](https://mlsanity.com/paper/2205.12148/) (2022)
- [NorNE: Annotating Named Entities for Norwegian](https://mlsanity.com/paper/1911.12146/) (2019)
- [The Danish Gigaword Project](https://mlsanity.com/paper/2005.03521/) (2020)
- [Modelling Latent Translations for Cross-Lingual Transfer](https://mlsanity.com/paper/2107.11353/) (2021)
- [Cross-lingual Transfer of Monolingual Models](https://mlsanity.com/paper/2109.07348/) (2021)

## Built on by (most cited first, 15 of 15)

- [Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks](https://mlsanity.com/paper/2406.13469/) (2024)
- [M-IFEval: Multilingual Instruction-Following Evaluation](https://mlsanity.com/paper/2502.04688/) (2025)
- [CreoleVal: Multilingual Multitask Benchmarks for Creoles](https://mlsanity.com/paper/2310.19567/) (2023)
- [Targeted Multilingual Adaptation for Low-resource Language Families](https://mlsanity.com/paper/2405.12413/) (2024)
- [GradSim: Gradient-Based Language Grouping for Effective Multilingual Training](https://mlsanity.com/paper/2310.15269/) (2023)
- [FoQA: A Faroese Question-Answering Dataset](https://mlsanity.com/paper/2502.07642/) (2025)
- [Add Noise, Tasks, or Layers? MaiNLP at the VarDial 2025 Shared Task on Norwegian Dialectal Slot and Intent Detection](https://mlsanity.com/paper/2501.03870/) (2025)
- [Multi-label Scandinavian Language Identification (SLIDE)](https://mlsanity.com/paper/2502.06692/) (2025)
- [Recipe for Zero-shot POS Tagging: Is It Useful in Realistic Scenarios?](https://mlsanity.com/paper/2410.10576/) (2024)
- [Structural priors for data-efficient language learning](https://mlsanity.com/paper/2609.11505/) (2026)
- [DANSK and DaCy 2.6.0: Domain Generalization of Danish Named Entity Recognition](https://mlsanity.com/paper/2402.18209/) (2024)
- [MaiNLP at SemEval-2024 Task 1: Analyzing Source Language Selection in Cross-Lingual Textual Relatedness](https://mlsanity.com/paper/2404.02570/) (2024)
- [textscCantoNLU: A benchmark for Cantonese natural language understanding](https://mlsanity.com/paper/2510.20670/) (2025)
- [TiME: Tiny Monolingual Encoders for Efficient NLP Pipelines](https://mlsanity.com/paper/2512.14645/) (2025)
- [Harnessing Linguistic Dissimilarity for Language Generalization on Unseen Low-Resource Varieties](https://mlsanity.com/paper/2605.04500/) (2026)

## Abstract (arXiv)

Multilingual language models have pushed state-of-the-art in cross-lingual NLP transfer. The majority of zero-shot cross-lingual transfer, however, use one and the same massively multilingual transformer (e.g., mBERT or XLM-R) to transfer to all target languages, irrespective of their typological, etymological, and phylogenetic relations to other languages. In particular, readily available data and models of resource-rich sibling languages are often ignored. In this work, we empirically show, in a case study for Faroese -- a low-resource language from a high-resource language family -- that by leveraging the phylogenetic information and departing from the 'one-size-fits-all' paradigm, one can improve cross-lingual transfer to low-resource languages. In particular, we leverage abundant resources of other Scandinavian languages (i.e., Danish, Norwegian, Swedish, and Icelandic) for the benefit of Faroese. Our evaluation results show that we can substantially improve the transfer performance to Faroese by exploiting data and models of closely-related high-resource languages. Further, we release a new web corpus of Faroese and Faroese datasets for named entity recognition (NER), semantic text similarity (STS), and new language models trained on all Scandinavian languages.