# Multi-Head Adapter Routing for Cross-Task Generalization

arXiv: https://arxiv.org/abs/2211.03831 · HTML: https://mlsanity.com/paper/2211.03831/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 7 Nov 2022.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://mlsanity.com/paper/2110.07904/) (2021)
- [Recurrent Independent Mechanisms](https://mlsanity.com/paper/1909.10893/) (2019)
- [Training Neural Networks with Fixed Sparse Masks](https://mlsanity.com/paper/2111.09839/) (2021)
- [Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models](https://mlsanity.com/paper/2010.05874/) (2020)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [Composable Sparse Fine-Tuning for Cross-Lingual Transfer](https://mlsanity.com/paper/2110.07560/) (2021)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [Exploring and Predicting Transferability across NLP Tasks](https://mlsanity.com/paper/2005.00770/) (2020)
- [ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts](https://mlsanity.com/paper/2205.11961/) (2022)
- [Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing](https://mlsanity.com/paper/1807.00914/) (2018)
- [Continual Learning via Local Module Composition](https://mlsanity.com/paper/2111.07736/) (2021)
- [Modular Networks: Learning to Decompose Neural Computation](https://mlsanity.com/paper/1811.05249/) (2018)
- [IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages](https://mlsanity.com/paper/2201.11732/) (2022)
- [Routing Networks and the Challenges of Modular and Compositional Computation](https://mlsanity.com/paper/1904.12774/) (2019)

## Built on by (most cited first, 30 of 33)

- [Efficient Large Language Models: A Survey](https://mlsanity.com/paper/2312.03863/) (2023)
- [Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning](https://mlsanity.com/paper/2309.05444/) (2023)
- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [Learning to Route Among Specialized Experts for Zero-Shot Generalization](https://mlsanity.com/paper/2402.05859/) (2024)
- [VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks](https://mlsanity.com/paper/2405.15179/) (2024)
- [A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning](https://mlsanity.com/paper/2408.07057/) (2024)
- [Multi-Head Mixture-of-Experts](https://mlsanity.com/paper/2404.15045/) (2024)
- [Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization](https://mlsanity.com/paper/2311.09344/) (2023)
- [ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization](https://mlsanity.com/paper/2311.13171/) (2023)
- [Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?](https://mlsanity.com/paper/2403.01929/) (2024)
- [Does Combining Parameter-efficient Modules Improve Few-shot Transfer Accuracy?](https://mlsanity.com/paper/2402.15414/) (2024)
- [PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model](https://mlsanity.com/paper/2411.08212/) (2024)
- [LoRA-Augmented Generation (LAG) for Knowledge-Intensive Language Tasks](https://mlsanity.com/paper/2507.05346/) (2025)
- [Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules](https://mlsanity.com/paper/2508.02587/) (2025)
- [Multi-Task Vehicle Routing Solver via Mixture of Specialized Experts under State-Decomposable MDP](https://mlsanity.com/paper/2510.21453/) (2025)
- [Simple Drop-in LoRA Conditioning on Attention Layers Will Improve Your Diffusion Model](https://mlsanity.com/paper/2405.03958/) (2024)
- [SpectR: Dynamically Composing LM Experts with Spectral Routing](https://mlsanity.com/paper/2504.03454/) (2025)
- [Parametric Retrieval-Augmented Generation using Latent Routing of LoRA Adapters](https://mlsanity.com/paper/2511.17044/) (2025)
- [Glider: Global and Local Instruction-Driven Expert Router](https://mlsanity.com/paper/2410.07172/) (2024)
- [One Rank at a Time: Cascading Error Dynamics in Sequential Learning](https://mlsanity.com/paper/2505.22602/) (2025)
- [FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts](https://mlsanity.com/paper/2506.00495/) (2025)
- [OrchMoE: Efficient Multi-Adapter Learning with Task-Skill Synergy](https://mlsanity.com/paper/2401.10559/) (2024)
- [Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models](https://mlsanity.com/paper/2408.15915/) (2024)
- [AdaptGCD: Multi-Expert Adapter Tuning for Generalized Category Discovery](https://mlsanity.com/paper/2410.21705/) (2024)
- [Generative Modeling of Individual Behavior at Scale](https://mlsanity.com/paper/2502.14998/) (2025)
- [Adapting Time Series Foundation Models through Data Mixtures](https://mlsanity.com/paper/2603.02840/) (2026)
- [Parametric Memory Decoding for Zero-Shot Routing in LoRA-Based External Parametric Memory](https://mlsanity.com/paper/2607.04118/) (2026)
- [One Arrow, Many Targets: Probing LLMs for Multi-Attribute Controllable Text Summarization](https://mlsanity.com/paper/2411.01213/) (2024)
- [SEQR: Secure and Efficient QR-based LoRA Routing](https://mlsanity.com/paper/2509.18093/) (2025)
- [Fine-Grained DINO Tuning with Dual Supervision for Face Forgery Detection](https://mlsanity.com/paper/2511.12107/) (2025)

## Abstract (arXiv)

Parameter-efficient fine-tuning (PEFT) for cross-task generalization consists in pre-training adapters on a multi-task training set before few-shot adaptation to test tasks. Polytropon [Ponti et al., 2023] (textttPoly) jointly learns an inventory of adapters and a routing function that selects a (variable-size) subset of adapters for each task during both pre-training and few-shot adaptation. In this paper, we investigate the role that adapter routing plays in its success and design new variants based on our findings. First, we build on the intuition that finer-grained routing provides more expressivity. Hence, we propose textttMHR (Multi-Head Routing) which combines subsets of adapter parameters and outperforms textttPoly under a comparable parameter budget; by only fine-tuning the routing function and not the adapters (textttMHR-z) we achieve competitive performance with extreme parameter efficiency. Second, we find that textttPoly/textttMHR performance is a result of better multi-task optimization, rather than modular inductive biases that facilitate adapter recombination and local adaptation, as previously hypothesized. In fact, we find that textttMHR exhibits high gradient alignment between training tasks. We find that routing is most beneficial during multi-task pre-training rather than during few-shot adaptation and propose textttMHR-μ, which discards routing and fine-tunes the average of the pre-trained adapters on each downstream tasks. This establishes textttMHR-μ as an effective method for single-adapter fine-tuning. We also show that textttMHR-μ can be used as an effective zero-shot transfer method by training the average of the pre-trained adapters for a few additional steps on the multi-task training set: this yields gains up to 3% on absolute accuracy w.r.t. the baselines.