mlsanityData to 7 Oct 2026

HyperPrompt: Prompt-based Task-Conditioning of Transformers

arXiv 2203.00759 ↗First posted 1 Mar 2022cs.CL cs.LGcited by 67

Not on the map. Why?

Abstract (arXiv)

Prompt-Tuning is a new paradigm for finetuning pre-trained language models in a parameter-efficient way. Here, we explore the use of HyperNetworks to generate hyper-prompts: we propose HyperPrompt, a novel architecture for prompt-based task-conditioning of self-attention in Transformers. The hyper-prompts are end-to-end learnable via generation by a HyperNetwork. HyperPrompt allows the network to learn task-specific feature maps where the hyper-prompts serve as task global memories for the queries to attend to, at the same time enabling flexible information sharing among tasks. We show that HyperPrompt is competitive against strong multi-task learning baselines with as few as 0.14% of additional task-conditioning parameters, achieving great parameter and computational efficiency. Through extensive empirical experiments, we demonstrate that HyperPrompt can achieve superior performances over strong T5 multi-task learning baselines and parameter-efficient adapter variants including Prompt-Tuning and HyperFormer++ on Natural Language Understanding benchmarks of GLUE and SuperGLUE across many model sizes.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Builds on 31 of its 36 references that are in the map. First posted on arXiv 1 Mar 2022.

Builds on

31 of 36 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
19 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  3. 2014How transferable are features in deep neural networks?cited by 1,216
  4. 2016HyperNetworkscited by 1,003
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  6. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  7. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  8. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  9. 2019Continual learning with hypernetworkscited by 267
  10. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  11. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  12. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  13. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  14. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  15. 2020Understanding and Improving Information Transfer in Multi-Task Learningcited by 113
  16. 2021The Efficiency Misnomercited by 98
  17. 2019Augmenting Self-attention with Persistent Memorycited by 92
  18. 2020Small Towers Make Big Differencescited by 7
  19. 2020HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projectionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 22; 2024: 16; 2025: 11; 2026: 3.

67 papers.

Sort:

By area:

  1. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  2. 2022UL2: Unifying Language Learning Paradigmscited by 292
  3. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  4. 2023PromptIR: Prompting for All-in-One Blind Image Restorationcited by 113
  5. 2023Modular Deep Learningcited by 82
  6. 2023E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuningcited by 70
  7. 2022Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learningcited by 57
  8. 2023Prompt Engineering for Healthcare: Methodologies and Applicationscited by 52
  9. 2022Bayesian Prompt Learning for Image-Language Model Generalizationcited by 51
  10. 2022TEMPERA: Test-Time Prompting via Reinforcement Learningcited by 50
  11. 2025Text-to-LoRA: Instant Transformer Adaptioncited by 50
  12. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 45
38 more
  1. 2024Global and Local Prompts Cooperation via Optimal Transport for Federated Learningcited by 45
  2. 2022Two-stage LLM Fine-tuning with Less Specialization and More Generalizationcited by 39
  3. 2023Batched Low-Rank Adaptation of Foundation Modelscited by 32
  4. 2024Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?cited by 31
  5. 2022XPrompt: Exploring the Extreme of Prompt Tuningcited by 30
  6. 2024Visual Fourier Prompt Tuningcited by 30
  7. 2023InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understandingcited by 29
  8. 2022HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisationcited by 28
  9. 2024HyperMoE: Towards Better Mixture of Experts via Transferring Among Expertscited by 25
  10. 2022Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfercited by 22
  11. 2022Hyperdecoders: Instance-specific decoders for multi-task NLPcited by 21
  12. 2022Prompt Tuning for Parameter-efficient Medical Image Segmentationcited by 21
  13. 2024M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learningcited by 16
  14. 2024Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognitioncited by 16
  15. 2024Meta-Task Prompting Elicits Embeddings from Large Language Modelscited by 15
  16. 2025Instant Personalized Large Language Model Adaptation via Hypernetworkcited by 15
  17. 2023Effective Structured Prompting by Meta-Learning and Representative Verbalizercited by 14
  18. 2024PromptRR: Diffusion Models as Prompt Generators for Single Image Reflection Removalcited by 14
  19. 2023Prompt Guided Transformer for Multi-Task Dense Predictioncited by 13
  20. 2022HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Taskscited by 12
  21. 2023Focus Your Attention (with Adaptive IIR Filters)cited by 12
  22. 2024A Survey of Controllable Learning: Methods and Applications in Information Retrievalcited by 12
  23. 2023Task-Agnostic Low-Rank Adapters for Unseen English Dialectscited by 9
  24. 2022Task Bias in Vision-Language Modelscited by 8
  25. 2023Visual Exemplar Driven Task-Prompting for Unified Perception in Autonomous Drivingcited by 8
  26. 2023Hierarchical Side-Tuning for Vision Transformerscited by 8
  27. 2023Decomposed Prompt Tuning via Low-Rank Reparameterizationcited by 8
  28. 2024C2A: Client-Customized Adaptation for Parameter-Efficient Federated Learningcited by 8
  29. 2025Foundation Models and Transformers for Anomaly Detection: A Surveycited by 8
  30. 2023Prompter: Zero-shot Adaptive Prefixes for Dialogue State Tracking Domain Adaptationcited by 7
  31. 2024PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learningcited by 7
  32. 2024Decoupling Weighing and Selecting for Integrating Multiple Graph Pre-training Taskscited by 7
  33. 2023PromptAttack: Probing Dialogue State Trackers with Adversarial Promptscited by 6
  34. 2023Developing Effective Educational Chatbots with ChatGPT prompts: Insights from Preliminary Tests in a Case Study on Social Media Literacy (with appendix)cited by 6
  35. 2023Prototype-based HyperAdapter for Sample-Efficient Multi-task Tuningcited by 6
  36. 2025FOCUS: Fused Observation of Channels for Unveiling Spectracited by 6
  37. 2023Full Scaling Automation for Sustainable Development of Green Data Centerscited by 5
  38. 2023Flatness-Aware Prompt Selection Improves Accuracy and Sample Efficiencycited by 5

50 most cited of 67 All 67 (JSON)