# HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation

arXiv: https://arxiv.org/abs/2212.10315 · HTML: https://mlsanity.com/paper/2212.10315/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 20 Dec 2022.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?](https://mlsanity.com/paper/2204.05832/) (2022)
- [Learning from Task Descriptions](https://mlsanity.com/paper/2011.08115/) (2020)
- [HyperPrompt: Prompt-based Task-Conditioning of Transformers](https://mlsanity.com/paper/2203.00759/) (2022)
- [HyperTuning: Toward Adapting Large Language Models without Back-propagation](https://mlsanity.com/paper/2211.12485/) (2022)
- [Contextual Parameter Generation for Universal Neural Machine Translation](https://mlsanity.com/paper/1808.08493/) (2018)
- [Learning to Generate Task-Specific Adapters from Task Description](https://mlsanity.com/paper/2101.00420/) (2021)
- [Hyperdecoders: Instance-specific decoders for multi-task NLP](https://mlsanity.com/paper/2203.08304/) (2022)
- [Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer](https://mlsanity.com/paper/2205.12148/) (2022)

## Built on by (most cited first, 28 of 28)

- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [FireAct: Toward Language Agent Fine-tuning](https://mlsanity.com/paper/2310.05915/) (2023)
- [Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning](https://mlsanity.com/paper/2309.05444/) (2023)
- [Text-to-LoRA: Instant Transformer Adaption](https://mlsanity.com/paper/2506.06105/) (2025)
- [Doc-to-LoRA: Learning to Instantly Internalize Contexts](https://mlsanity.com/paper/2602.15902/) (2026)
- [Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models](https://mlsanity.com/paper/2508.09968/) (2025)
- [Large Language Model Instruction Following: A Survey of Progresses and Challenges](https://mlsanity.com/paper/2303.10475/) (2023)
- [HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts](https://mlsanity.com/paper/2402.12656/) (2024)
- [Instant Personalized Large Language Model Adaptation via Hypernetwork](https://mlsanity.com/paper/2510.16282/) (2025)
- [Supervised Fine-tuning in turn Improves Visual Foundation Models](https://mlsanity.com/paper/2401.10222/) (2024)
- [ModelGPT: Unleashing LLM's Capabilities for Tailored Model Generation](https://mlsanity.com/paper/2402.12408/) (2024)
- [Effective Heterogeneous Federated Learning via Efficient Hypernetwork-based Weight Generation](https://mlsanity.com/paper/2407.03086/) (2024)
- [A Survey on Unlearnable Data](https://mlsanity.com/paper/2503.23536/) (2025)
- [Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions](https://mlsanity.com/paper/2506.11111/) (2025)
- [HyperLoRA for PDEs](https://mlsanity.com/paper/2308.09290/) (2023)
- [Investigating the Effectiveness of HyperTuning via Gisting](https://mlsanity.com/paper/2402.16817/) (2024)
- [HyperAlign: Hypernetwork for Efficient Test-Time Alignment of Diffusion Models](https://mlsanity.com/paper/2601.15968/) (2026)
- [HAPS: Hierarchical LLM Routing with Joint Architecture and Parameter Search](https://mlsanity.com/paper/2601.05903/) (2026)
- [Doc-to-Atom: Learning to Compile and Compose Memory Atoms](https://mlsanity.com/paper/2606.12400/) (2026)
- [On the Transformations across Reward Model, Parameter Update, and In-Context Prompt](https://mlsanity.com/paper/2406.16377/) (2024)
- [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://mlsanity.com/paper/2607.02512/) (2026)
- [PhaseLoRA: Control-Regime-Conditioned Low-Rank Adaptation for Continuous-Action Vision-Language-Action Policies](https://mlsanity.com/paper/2608.15285/) (2026)
- [Shuttle Between the Instructions and the Parameters of Large Language Models](https://mlsanity.com/paper/2502.02315/) (2025)
- [Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones](https://mlsanity.com/paper/2506.05641/) (2025)
- [PRISMR: Overcoming Parse Collapse in Multimodal Listwise Ranking via Parameterized Representation Internalization](https://mlsanity.com/paper/2606.12942/) (2026)
- [HyperStyler: Low-resource Authorship Style Transfer via Context-aware Style Navigation and Hypernetworks](https://mlsanity.com/paper/2609.02772/) (2026)
- [LoRA-generating hypernetworks for efficient on-device LLM generative personalization](https://mlsanity.com/paper/2609.24979/) (2026)
- [Learning to Predict Distributions over Weight Updates for Test-Time Adaptation](https://mlsanity.com/paper/2610.01934/) (2026)

## Abstract (arXiv)

Recent NLP models have shown the remarkable ability to effectively generalise `zero-shot' to new tasks using only natural language instructions as guidance. However, many of these approaches suffer from high computational costs due to their reliance on concatenating lengthy instructions with every input example, resulting in costly reprocessing of the instruction. To avoid this, we introduce Hypernetworks for INstruction Tuning (HINT), which convert task instructions and examples into parameter-efficient modules inserted into an underlying model using a pretrained text encoder, eliminating the need to include instructions in the model input. The hypernetwork in HINT also produces an encoded instruction, which we concatenate with encoded inputs during decoding to further improve performance. HINT models outperform strong state-of-the-art baselines by over 10% when controlling for compute (measured in FLOPs). By converting instructions into modules, HINT models can effectively disregard the length of instructions and few-shot example inputs in terms of compute usage. As a result, HINT can enhance its performance by up to 25% by incorporating additional few-shot data, while utilizing only up to 5% more compute. This combines the strengths of parameter-efficient fine-tuning and in-context learning.