# Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer

arXiv: https://arxiv.org/abs/2012.06460 · HTML: https://mlsanity.com/paper/2012.06460/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2020.

## Built on by (most cited first, 23 of 23)

- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://mlsanity.com/paper/2205.06266/) (2022)
- [Composable Sparse Fine-Tuning for Cross-Lingual Transfer](https://mlsanity.com/paper/2110.07560/) (2021)
- [Sustainable Modular Debiasing of Language Models](https://mlsanity.com/paper/2109.03646/) (2021)
- [xGQA: Cross-Lingual Visual Question Answering](https://mlsanity.com/paper/2109.06082/) (2021)
- [What to Pre-Train on? Efficient Intermediate Task Selection](https://mlsanity.com/paper/2104.08247/) (2021)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization](https://mlsanity.com/paper/2311.09344/) (2023)
- [mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations](https://mlsanity.com/paper/2305.14224/) (2023)
- [Specializing Multilingual Language Models: An Empirical Study](https://mlsanity.com/paper/2106.09063/) (2021)
- [Cross-Lingual Transfer with Target Language-Ready Task Adapters](https://mlsanity.com/paper/2306.02767/) (2023)
- [Unsupervised Domain Adaptation with Adapter](https://mlsanity.com/paper/2111.00667/) (2021)
- [Fine-Tuning BERT with Character-Level Noise for Zero-Shot Transfer to Dialects and Closely-Related Languages](https://mlsanity.com/paper/2303.17683/) (2023)
- [AdapterHub Playground: Simple and Flexible Few-Shot Learning with Adapters](https://mlsanity.com/paper/2108.08103/) (2021)
- [Accelerating Multilingual Language Model for Excessively Tokenized Languages](https://mlsanity.com/paper/2401.10660/) (2024)
- [Distilling Efficient Language-Specific Models for Cross-Lingual Transfer](https://mlsanity.com/paper/2306.01709/) (2023)
- [Efficient Domain Adaptation of Sentence Embeddings Using Adapters](https://mlsanity.com/paper/2307.03104/) (2023)
- [Target-Oriented Fine-tuning for Zero-Resource Named Entity Recognition](https://mlsanity.com/paper/2107.10523/) (2021)
- [Legal and Political Stance Detection of SCOTUS Language](https://mlsanity.com/paper/2211.11724/) (2022)
- [Cross-Lingual Text Classification with Multilingual Distillation and Zero-Shot-Aware Training](https://mlsanity.com/paper/2202.13654/) (2022)
- [Inducer-tuning: Connecting Prefix-tuning and Adapter-tuning](https://mlsanity.com/paper/2210.14469/) (2022)
- [Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fast](https://mlsanity.com/paper/2305.11449/) (2023)
- [Feature Aggregation in Zero-Shot Cross-Lingual Transfer Using Multilingual BERT](https://mlsanity.com/paper/2205.08497/) (2022)
- [How to Tune a Multilingual Encoder Model for Germanic Languages: A Study of PEFT, Full Fine-Tuning, and Language Adapters](https://mlsanity.com/paper/2501.06025/) (2025)

## Abstract (arXiv)

Adapter modules, additional trainable parameters that enable efficient fine-tuning of pretrained transformers, have recently been used for language specialization of multilingual transformers, improving downstream zero-shot cross-lingual transfer. In this work, we propose orthogonal language and task adapters (dubbed orthoadapters) for cross-lingual transfer. They are trained to encode language- and task-specific information that is complementary (i.e., orthogonal) to the knowledge already stored in the pretrained transformer's parameters. Our zero-shot cross-lingual transfer experiments, involving three tasks (POS-tagging, NER, NLI) and a set of 10 diverse languages, 1) point to the usefulness of orthoadapters in cross-lingual transfer, especially for the most complex NLI task, but also 2) indicate that the optimal adapter configuration highly depends on the task and the target language. We hope that our work will motivate a wider investigation of usefulness of orthogonality constraints in language- and task-specific fine-tuning of pretrained transformers.