# Learning to Generate Task-Specific Adapters from Task Description

arXiv: https://arxiv.org/abs/2101.00420 · HTML: https://mlsanity.com/paper/2101.00420/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 2 Jan 2021.

## Builds on (most cited first, 21 of 21)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [A Unified MRC Framework for Named Entity Recognition](https://mlsanity.com/paper/1910.11476/) (2019)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [Zero-Shot Entity Linking by Reading Entity Descriptions](https://mlsanity.com/paper/1906.07348/) (2019)
- [Learning from Task Descriptions](https://mlsanity.com/paper/2011.08115/) (2020)
- [UDapter: Language Adaptation for Truly Universal Dependency Parsing](https://mlsanity.com/paper/2004.14327/) (2020)

## Built on by (most cited first, 28 of 28)

- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2404.15522/) (2024)
- [HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learning](https://mlsanity.com/paper/2201.04182/) (2022)
- [Exploiting Adapters for Cross-lingual Low-resource Speech Recognition](https://mlsanity.com/paper/2105.11905/) (2021)
- [Robustness of Learning from Task Instructions](https://mlsanity.com/paper/2212.03813/) (2022)
- [Large Language Model Instruction Following: A Survey of Progresses and Challenges](https://mlsanity.com/paper/2303.10475/) (2023)
- [IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization](https://mlsanity.com/paper/2411.06208/) (2024)
- [HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation](https://mlsanity.com/paper/2212.10315/) (2022)
- [InstructABSA: Instruction Learning for Aspect Based Sentiment Analysis](https://mlsanity.com/paper/2302.08624/) (2023)
- [Hyperdecoders: Instance-specific decoders for multi-task NLP](https://mlsanity.com/paper/2203.08304/) (2022)
- [Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policies](https://mlsanity.com/paper/2310.16686/) (2023)
- [In-BoXBART: Get Instructions into Biomedical Multi-Task Learning](https://mlsanity.com/paper/2204.07600/) (2022)
- [Multilingual Machine Translation with Hyper-Adapters](https://mlsanity.com/paper/2205.10835/) (2022)
- [MEND: Meta dEmonstratioN Distillation for Efficient and Effective In-Context Learning](https://mlsanity.com/paper/2403.06914/) (2024)
- [Context-NER : Contextual Phrase Generation at Scale](https://mlsanity.com/paper/2109.08079/) (2021)
- [EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Models](https://mlsanity.com/paper/2506.08375/) (2025)
- [Hire Me or Not? Examining Language Model's Behavior with Occupation Attributes](https://mlsanity.com/paper/2405.06687/) (2024)
- [From Instance Training to Instruction Learning: Task Adapters Generation from Instructions](https://mlsanity.com/paper/2406.12382/) (2024)
- [EDM3: Event Detection as Multi-task Text Generation](https://mlsanity.com/paper/2305.16357/) (2023)
- [Meta-Ensemble Parameter Learning](https://mlsanity.com/paper/2210.01973/) (2022)
- [AdaptGCD: Multi-Expert Adapter Tuning for Generalized Category Discovery](https://mlsanity.com/paper/2410.21705/) (2024)
- [Generative Meta-Learning for Zero-Shot Relation Triplet Extraction](https://mlsanity.com/paper/2305.01920/) (2023)
- [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://mlsanity.com/paper/2607.02512/) (2026)
- [Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones](https://mlsanity.com/paper/2506.05641/) (2025)
- [TeNet: Text-to-Network for Compact Policy Synthesis](https://mlsanity.com/paper/2601.15912/) (2026)
- [Amortized Low-Rank Adaptation for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2609.12278/) (2026)
- [Doc2LoRA Provides Decodable Representations of Scientific Ideas](https://mlsanity.com/paper/2609.38374/) (2026)

## Abstract (arXiv)

Pre-trained text-to-text transformers such as BART have achieved impressive performance across a range of NLP tasks. Recent study further shows that they can learn to generalize to novel tasks, by including task descriptions as part of the source sequence and training the model with (source, target) examples. At test time, these fine-tuned models can make inferences on new tasks using the new task descriptions as part of the input. However, this approach has potential limitations, as the model learns to solve individual (source, target) examples (i.e., at the instance level), instead of learning to solve tasks by taking all examples within a task as a whole (i.e., at the task level). To this end, we introduce Hypter, a framework that improves text-to-text transformer's generalization ability to unseen tasks by training a hypernetwork to generate task-specific, light-weight adapters from task descriptions. Experiments on ZEST dataset and a synthetic SQuAD dataset demonstrate that Hypter improves upon fine-tuning baselines. Notably, when using BART-Large as the main network, Hypter brings 11.3% comparative improvement on ZEST dataset.