mlsanityData to 7 Oct 2026

Learning to Generate Task-Specific Adapters from Task Description

arXiv 2101.00420 ↗First posted 2 Jan 2021cs.CL cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

Pre-trained text-to-text transformers such as BART have achieved impressive performance across a range of NLP tasks. Recent study further shows that they can learn to generalize to novel tasks, by including task descriptions as part of the source sequence and training the model with (source, target) examples. At test time, these fine-tuned models can make inferences on new tasks using the new task descriptions as part of the input. However, this approach has potential limitations, as the model learns to solve individual (source, target) examples (i.e., at the instance level), instead of learning to solve tasks by taking all examples within a task as a whole (i.e., at the task level). To this end, we introduce Hypter, a framework that improves text-to-text transformer's generalization ability to unseen tasks by training a hypernetwork to generate task-specific, light-weight adapters from task descriptions. Experiments on ZEST dataset and a synthetic SQuAD dataset demonstrate that Hypter improves upon fine-tuning baselines. Notably, when using BART-Large as the main network, Hypter brings 11.3% comparative improvement on ZEST dataset.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 21 of its 27 references that are in the map. First posted on arXiv 2 Jan 2021.

Builds on

21 of 27 references.

References · 21

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  8. 2016HyperNetworkscited by 1,003
  9. 2019How Can We Know What Language Models Know?cited by 953
  10. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  11. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  12. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
9 more
  1. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  2. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  3. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  4. 2019Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approachcited by 299
  5. 2019A Unified MRC Framework for Named Entity Recognitioncited by 199
  6. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
  7. 2019Zero-Shot Entity Linking by Reading Entity Descriptionscited by 124
  8. 2020Learning from Task Descriptionscited by 78
  9. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 9; 2023: 5; 2024: 6; 2025: 2; 2026: 4.

28 papers.

Sort:

By area:

  1. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  2. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  3. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Modelscited by 67
  4. 2022HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learningcited by 55
  5. 2022Robustness of Learning from Task Instructionscited by 34
  6. 2021Exploiting Adapters for Cross-lingual Low-resource Speech Recognitioncited by 33
  7. 2023Large Language Model Instruction Following: A Survey of Progresses and Challengescited by 31
  8. 2024IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimizationcited by 30
  9. 2022HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisationcited by 28
  10. 2023InstructABSA: Instruction Learning for Aspect Based Sentiment Analysiscited by 22
  11. 2022Hyperdecoders: Instance-specific decoders for multi-task NLPcited by 21
  12. 2023Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policiescited by 18
16 more
  1. 2022In-BoXBART: Get Instructions into Biomedical Multi-Task Learningcited by 16
  2. 2022Multilingual Machine Translation with Hyper-Adapterscited by 15
  3. 2024MEND: Meta dEmonstratioN Distillation for Efficient and Effective In-Context Learningcited by 15
  4. 2021Context-NER : Contextual Phrase Generation at Scalecited by 11
  5. 2025EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Modelscited by 9
  6. 2024Hire Me or Not? Examining Language Model's Behavior with Occupation Attributescited by 5
  7. 2024From Instance Training to Instruction Learning: Task Adapters Generation from Instructionscited by 4
  8. 2023EDM3: Event Detection as Multi-task Text Generationcited by 3
  9. 2022Meta-Ensemble Parameter Learningcited by 2
  10. 2024AdaptGCD: Multi-Expert Adapter Tuning for Generalized Category Discoverycited by 2
  11. 2023Generative Meta-Learning for Zero-Shot Relation Triplet Extractioncited by 1
  12. 2026Program-as-Weights: A Programming Paradigm for Fuzzy Functionscited by 1
  13. 2025Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Onescited by 0
  14. 2026TeNet: Text-to-Network for Compact Policy Synthesiscited by 0
  15. 2026Amortized Low-Rank Adaptation for Model-Based Reinforcement Learningcited by 0
  16. 2026Doc2LoRA Provides Decodable Representations of Scientific Ideascited by 0