# Example-based Hypernetworks for Out-of-Distribution Generalization

arXiv: https://arxiv.org/abs/2203.14276 · HTML: https://mlsanity.com/paper/2203.14276/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 71 references that are in the map.
- First posted on arXiv 27 Mar 2022.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning](https://mlsanity.com/paper/1903.10258/) (2019)
- [Personalized Federated Learning using Hypernetworks](https://mlsanity.com/paper/2103.04628/) (2021)
- [Neural Unsupervised Domain Adaptation in NLP---A Survey](https://mlsanity.com/paper/2006.00632/) (2020)
- [Distributionally Robust Language Modeling](https://mlsanity.com/paper/1909.02060/) (2019)
- [On Calibration and Out-of-domain Generalization](https://mlsanity.com/paper/2102.10395/) (2021)
- [UDapter: Language Adaptation for Truly Universal Dependency Parsing](https://mlsanity.com/paper/2004.14327/) (2020)
- [Multi-source Distilling Domain Adaptation](https://mlsanity.com/paper/1911.11554/) (2019)
- [Bayesian Hypernetworks](https://mlsanity.com/paper/1710.04759/) (2017)
- [Hypernetwork functional image representation](https://mlsanity.com/paper/1902.10404/) (2019)
- [Multi-Source Domain Adaptation with Mixture of Experts](https://mlsanity.com/paper/1809.02256/) (2018)
- [PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains](https://mlsanity.com/paper/2102.12206/) (2021)
- [Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks](https://mlsanity.com/paper/1911.03863/) (2019)
- [Hyper-Graph-Network Decoders for Block Codes](https://mlsanity.com/paper/1909.09036/) (2019)
- [Contextual Parameter Generation for Universal Neural Machine Translation](https://mlsanity.com/paper/1808.08493/) (2018)
- [DoCoGen: Domain Counterfactual Generation for Low Resource Domain Adaptation](https://mlsanity.com/paper/2202.12350/) (2022)
- [PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models](https://mlsanity.com/paper/2006.09075/) (2020)

## Built on by (most cited first, 16 of 16)

- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [A Brief Review of Hypernetworks in Deep Learning](https://mlsanity.com/paper/2306.06955/) (2023)
- [HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation](https://mlsanity.com/paper/2212.10315/) (2022)
- [HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts](https://mlsanity.com/paper/2402.12656/) (2024)
- [Zero-shot Transfer of Article-aware Legal Outcome Classification for European Court of Human Rights Cases](https://mlsanity.com/paper/2302.00609/) (2023)
- [Domain Generalization in Machine Learning Models for Wireless Communications: Concepts, State-of-the-Art, and Open Issues](https://mlsanity.com/paper/2303.08106/) (2023)
- [Measuring the Robustness of NLP Models to Domain Shifts](https://mlsanity.com/paper/2306.00168/) (2023)
- [HMOE: Hypernetwork-based Mixture of Experts for Domain Generalization](https://mlsanity.com/paper/2211.08253/) (2022)
- [In-context Learning of Evolving Data Streams with Tabular Foundational Models](https://mlsanity.com/paper/2502.16840/) (2025)
- [IDANI: Inference-time Domain Adaptation via Neuron-level Interventions](https://mlsanity.com/paper/2206.00259/) (2022)
- [Hyper-Compression: Model Compression via Hyperfunction](https://mlsanity.com/paper/2409.00592/) (2024)
- [Learning to Forget using Hypernetworks](https://mlsanity.com/paper/2412.00761/) (2024)
- [Exploiting Task Relationships in Continual Learning via Transferability-Aware Task Embeddings](https://mlsanity.com/paper/2502.11609/) (2025)
- [Text2Model: Text-based Model Induction for Zero-shot Image Classification](https://mlsanity.com/paper/2210.15182/) (2022)
- [Domain Generalization Deep Graph Transformation](https://mlsanity.com/paper/2305.11389/) (2023)
- [Channel-Wise MLPs Improve the Generalization of Recurrent Convolutional Networks](https://mlsanity.com/paper/2508.08298/) (2025)

## Abstract (arXiv)

As Natural Language Processing (NLP) algorithms continually achieve new milestones, out-of-distribution generalization remains a significant challenge. This paper addresses the issue of multi-source adaptation for unfamiliar domains: We leverage labeled data from multiple source domains to generalize to unknown target domains at training. Our innovative framework employs example-based Hypernetwork adaptation: a T5 encoder-decoder initially generates a unique signature from an input example, embedding it within the source domains' semantic space. This signature is subsequently utilized by a Hypernetwork to generate the task classifier's weights. We evaluated our method across two tasks - sentiment classification and natural language inference - in 29 adaptation scenarios, where it outpaced established algorithms. In an advanced version, the signature also enriches the input example's representation. We also compare our finetuned architecture to few-shot GPT-3, demonstrating its effectiveness in essential use cases. To our knowledge, this marks the first application of Hypernetworks to the adaptation for unknown domains.