mlsanityData to 7 Oct 2026

Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks

arXiv 2106.04489 ↗First posted 8 Jun 2021cs.CLcited by 278

Not on the map. Why?

Abstract (arXiv)

State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. However, such modules are trained separately for each task and thus do not enable sharing information across tasks. In this paper, we show that we can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position, and layer id in a transformer model. This parameter-efficient multi-task learning framework allows us to achieve the best of both worlds by sharing knowledge across tasks via hypernetworks while enabling the model to adapt to each individual task through task-specific adapters. Experiments on the well-known GLUE benchmark show improved performance in multi-task learning while adding only 0.29% parameters per task. We additionally demonstrate substantial performance improvements in few-shot domain generalization across a variety of tasks. Our code is publicly available in https://github.com/rabeehk/hyperformer.

Key facts. Cited by 278 papers in the map (data through 7 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

36 of 50 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2016Layer Normalizationcited by 5,579
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
24 more
  1. 2016HyperNetworkscited by 1,003
  2. 2018Neural Network Acceptability Judgmentscited by 952
  3. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  4. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  5. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  6. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  7. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  8. 2019Continual learning with hypernetworkscited by 267
  9. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  10. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  11. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  12. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  13. 2018Characterizing and Avoiding Negative Transfercited by 226
  14. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  15. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  16. 2017Modulating early visual processing by languagecited by 167
  17. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  18. 2020Exploring and Predicting Transferability across NLP Taskscited by 126
  19. 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
  20. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77
  21. 2019Learning to Few-Shot Learn Across Diverse Natural Language Classification Taskscited by 57
  22. 2020Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Startcited by 44
  23. 2018Contextual Parameter Generation for Universal Neural Machine Translationcited by 40
  24. 2020Parameter Space Factorization for Zero-Shot Learning across Tasks and Languagescited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 51; 2023: 66; 2024: 79; 2025: 41; 2026: 29.

278 papers.

Sort:

By area:

  1. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  3. 2024Large Language Models: A Surveycited by 506
  4. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  5. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  6. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  7. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  8. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  9. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  10. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  11. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  12. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
38 more
  1. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  2. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  3. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
  4. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  5. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  6. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  7. 2023A Brief Review of Hypernetworks in Deep Learningcited by 102
  8. 2021The Efficiency Misnomercited by 98
  9. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  10. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  11. 2022Three things everyone should know about Vision Transformerscited by 90
  12. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  13. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  14. 2023Modular Deep Learningcited by 82
  15. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 80
  16. 2022StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuationcited by 74
  17. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  18. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  19. 2022CLiMB: A Continual Learning Benchmark for Vision-and-Language Taskscited by 68
  20. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  21. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  22. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  23. 2023VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understandingcited by 62
  24. 2021Multi-Task Learning in Natural Language Processing: An Overviewcited by 61
  25. 2022Supervised Training of Conditional Monge Mapscited by 60
  26. 2022HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learningcited by 55
  27. 2022Polyhistor: Parameter-Efficient Multi-Task Adaptation for Dense Vision Taskscited by 52
  28. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Bankscited by 52
  29. 2025Text-to-LoRA: Instant Transformer Adaptioncited by 50
  30. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 48
  31. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 45
  32. 2024MTLoRA: A Low-Rank Adaptation Approach for Efficient Multi-Task Learningcited by 44
  33. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  34. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
  35. 2022PERFECT: Prompt-free and Efficient Few-shot Learning with Language Modelscited by 40
  36. 2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40
  37. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  38. 2024OLoRA: Orthonormal Low-Rank Adaptation of Large Language Modelscited by 39

50 most cited of 278 All 278 (JSON)