mlsanityData to 14 Oct 2026

Example-based Hypernetworks for Out-of-Distribution Generalization

arXiv 2203.14276 ↗First posted 27 Mar 2022cs.CL cs.AI cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

As Natural Language Processing (NLP) algorithms continually achieve new milestones, out-of-distribution generalization remains a significant challenge. This paper addresses the issue of multi-source adaptation for unfamiliar domains: We leverage labeled data from multiple source domains to generalize to unknown target domains at training. Our innovative framework employs example-based Hypernetwork adaptation: a T5 encoder-decoder initially generates a unique signature from an input example, embedding it within the source domains' semantic space. This signature is subsequently utilized by a Hypernetwork to generate the task classifier's weights. We evaluated our method across two tasks - sentiment classification and natural language inference - in 29 adaptation scenarios, where it outpaced established algorithms. In an advanced version, the signature also enriches the input example's representation. We also compare our finetuned architecture to few-shot GPT-3, demonstrating its effectiveness in essential use cases. To our knowledge, this marks the first application of Hypernetworks to the adaptation for unknown domains.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 38 of its 71 references that are in the map. First posted on arXiv 27 Mar 2022.

Builds on

38 of 71 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  8. 2019Invariant Risk Minimizationcited by 1,854
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  10. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  11. 2016HyperNetworkscited by 1,027
  12. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 301
26 more
  1. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  2. 2019Continual learning with hypernetworkscited by 273
  3. 2019MetaPruning: Meta Learning for Automatic Neural Network Channel Pruningcited by 227
  4. 2021Personalized Federated Learning using Hypernetworkscited by 212
  5. 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 144
  6. 2019Distributionally Robust Language Modelingcited by 143
  7. 2021On Calibration and Out-of-domain Generalizationcited by 134
  8. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77
  9. 2019Multi-source Distilling Domain Adaptationcited by 73
  10. 2017Bayesian Hypernetworkscited by 66
  11. 2019Hypernetwork functional image representationcited by 65
  12. 2018Multi-Source Domain Adaptation with Mixture of Expertscited by 62
  13. 2021PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domainscited by 61
  14. 2019Learning to Few-Shot Learn Across Diverse Natural Language Classification Taskscited by 57
  15. 2019Hyper-Graph-Network Decoders for Block Codescited by 42
  16. 2018Contextual Parameter Generation for Universal Neural Machine Translationcited by 40
  17. 2022DoCoGen: Domain Counterfactual Generation for Low Resource Domain Adaptationcited by 37
  18. 2020PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Modelscited by 33
  19. 2017Implicit Weight Uncertainty in Neural Networkscited by 30
  20. 2019Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversioncited by 29
  21. 2019Deep Contextualized Self-training for Low Resource Dependency Parsingcited by 24
  22. 2020Transformer Based Multi-Source Domain Adaptationcited by 23
  23. 2016Neural Structural Correspondence Learning for Domain Adaptationcited by 19
  24. 2019Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domainscited by 13
  25. 2019A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networkscited by 12
  26. 2021DILBERT: Customized Pre-Training for Domain Adaptation withCategory Shift, with an Application to Aspect Extractioncited by 11

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 6; 2024: 3; 2025: 3.

16 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 359
  2. 2023A Brief Review of Hypernetworks in Deep Learningcited by 102
  3. 2022HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisationcited by 28
  4. 2024HyperMoE: Towards Better Mixture of Experts via Transferring Among Expertscited by 26
  5. 2023Zero-shot Transfer of Article-aware Legal Outcome Classification for European Court of Human Rights Casescited by 14
  6. 2023Domain Generalization in Machine Learning Models for Wireless Communications: Concepts, State-of-the-Art, and Open Issuescited by 12
  7. 2023Measuring the Robustness of NLP Models to Domain Shiftscited by 11
  8. 2022HMOE: Hypernetwork-based Mixture of Experts for Domain Generalizationcited by 8
  9. 2025In-context Learning of Evolving Data Streams with Tabular Foundational Modelscited by 8
  10. 2022IDANI: Inference-time Domain Adaptation via Neuron-level Interventionscited by 4
  11. 2024Hyper-Compression: Model Compression via Hyperfunctioncited by 3
  12. 2024Learning to Forget using Hypernetworkscited by 3
4 more
  1. 2025Exploiting Task Relationships in Continual Learning via Transferability-Aware Task Embeddingscited by 2
  2. 2022Text2Model: Text-based Model Induction for Zero-shot Image Classificationcited by 1
  3. 2023Domain Generalization Deep Graph Transformationcited by 1
  4. 2025Channel-Wise MLPs Improve the Generalization of Recurrent Convolutional Networkscited by 0