mlsanityData to 7 Oct 2026

HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learning

arXiv 2201.04182 ↗First posted 11 Jan 2022cs.LG cs.CVcited by 55

Not on the map. Why?

Abstract (arXiv)

In this work we propose a HyperTransformer, a Transformer-based model for supervised and semi-supervised few-shot learning that generates weights of a convolutional neural network (CNN) directly from support samples. Since the dependence of a small generated CNN model on a specific task is encoded by a high-capacity Transformer model, we effectively decouple the complexity of the large task space from the complexity of individual tasks. Our method is particularly effective for small target CNN architectures where learning a fixed universal task-independent embedding is not optimal and better performance is attained when the information about the task can modulate all model parameters. For larger models we discover that generating the last layer alone allows us to produce competitive or better results than those obtained with state-of-the-art methods while being end-to-end differentiable.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 11 Jan 2022.

Builds on

35 of 47 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  8. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  9. 2016Matching Networks for One Shot Learningcited by 2,246
  10. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  11. 2016HyperNetworkscited by 1,003
  12. 2018On First-Order Meta-Learning Algorithmscited by 1,003
23 more
  1. 2020Pre-Trained Image Processing Transformercited by 891
  2. 2018Meta-Learning with Latent Embedding Optimizationcited by 476
  3. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  4. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 435
  5. 2018TADAM: Task dependent adaptive metric for improved few-shot learningcited by 408
  6. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  7. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 346
  8. 2018How to train your MAMLcited by 306
  9. 2018Few-Shot Learning via Embedding Adaptation with Set-to-Set Functionscited by 293
  10. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  11. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  12. 2018Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learningcited by 208
  13. 2018Task-Agnostic Meta-Learning for Few-shot Learningcited by 151
  14. 2019Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processescited by 131
  15. 2017Few-Shot Image Recognition by Predicting Parameters from Activationscited by 122
  16. 2019Infinite Mixture Prototypes for Few-Shot Learningcited by 99
  17. 2020A Universal Representation Transformer Layer for Few-Shot Image Classificationcited by 65
  18. 2019HyperGAN: A Generative Model for Diverse, Performant Neural Networkscited by 51
  19. 2020Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Datacited by 50
  20. 2019Few-Shot Learning with Global Class Representationscited by 44
  21. 2021Learning to Generate Task-Specific Adapters from Task Descriptioncited by 28
  22. 2019LGM-Net: Learning to Generate Matching Networks for Few-Shot Learningcited by 27
  23. 2019PARN: Position-Aware Relation Networks for Few-Shot Learningcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 20; 2023: 12; 2024: 13; 2025: 9; 2026: 1.

55 papers.

Sort:

By area:

  1. 2022Transformers learn in-context by gradient descentcited by 714
  2. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  3. 2022Tutorial on amortized optimizationcited by 83
  4. 2022Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weightscited by 63
  5. 2022Model-Free Opponent Shapingcited by 43
  6. 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itselfcited by 40
  7. 2023MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learningcited by 40
  8. 2024Neural Network Diffusioncited by 39
  9. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 38
  10. 2024Diffusion-Based Neural Network Weights Generationcited by 38
  11. 2023Out-of-distributional risk bounds for neural operators with applications to the Helmholtz equationcited by 32
  12. 2023Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?cited by 29
38 more
  1. 2022HyperShot: Few-Shot Learning by Kernel HyperNetworkscited by 24
  2. 2022Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformercited by 22
  3. 2022Self-Promoted Supervision for Few-Shot Transformercited by 22
  4. 2024HyperFast: Instant Classification for Tabular Datacited by 22
  5. 2022Multi-Object Navigation with dynamically learned neural implicit representationscited by 21
  6. 2023MotherNet: Fast Training and Inference via Hyper-Network Transformerscited by 19
  7. 2022HyperNCA: Growing Developmental Networks with Neural Cellular Automatacited by 18
  8. 2023Towards Self-Assembling Artificial Neural Networks through Neural Developmental Programscited by 17
  9. 2022HyperMAML: Few-Shot Adaptation of Deep Models with Hypernetworkscited by 16
  10. 2022Few-Shot Learning Meets Transformer: Unified Query-Support Transformers for Few-Shot Classificationcited by 16
  11. 2023Deep Learning for Cross-Domain Few-Shot Visual Recognition: A Surveycited by 14
  12. 2022HyperMixer: An MLP-based Low Cost Alternative to Transformerscited by 13
  13. 2022Hyper-Representations for Pre-Training and Transfer Learningcited by 12
  14. 2024Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusioncited by 12
  15. 2023On the Importance of Spatial Relations for Few-shot Action Recognitioncited by 11
  16. 2024Meta-Exploiting Frequency Prior for Cross-Domain Few-Shot Learningcited by 11
  17. 2024Text-to-Model: Text-Conditioned Neural Network Diffusion for Train-Once-for-All Personalizationcited by 10
  18. 2022Meta-Learning via Classifier(-free) Diffusion Guidancecited by 9
  19. 2024When can transformers compositionally generalize in-context?cited by 7
  20. 2025Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstructioncited by 5
  21. 2023Continual HyperTransformer: A Meta-Learner for Continual Few-Shot Learningcited by 3
  22. 2025Instruction-Guided Autoregressive Neural Network Parameter Generationcited by 3
  23. 2022Meta-Ensemble Parameter Learningcited by 2
  24. 2023Federated Automatic Differentiationcited by 2
  25. 2024Unleash Graph Neural Networks from Heavy Tuningcited by 2
  26. 2024Enhancing Diversity in Bayesian Deep Learning via Hyperspherical Energy Minimization of CKAcited by 2
  27. 2025Few-shot Implicit Function Generation via Equivariancecited by 2
  28. 2025GeneralizeFormer: Layer-Adaptive Model Generation across Test-Time Distribution Shiftscited by 2
  29. 2025Hyper-Transforming Latent Diffusion Modelscited by 2
  30. 2023Generative Meta-Learning for Zero-Shot Relation Triplet Extractioncited by 1
  31. 2024Diffusing to the Top: Boost Graph Neural Networks with Minimal Hyperparameter Tuningcited by 1
  32. 2025Flows and Diffusions on the Neural Manifoldcited by 1
  33. 2025Semantic-guided LoRA Parameters Generationcited by 1
  34. 2022Expanding continual few-shot learning benchmarks to include recognition of specific instancescited by 0
  35. 2022Hypernetwork approach to Bayesian MAMLcited by 0
  36. 2023Evolution of Natural Language Processing Technology: Not Just Language Processing Towards General Purpose AIcited by 0
  37. 2023Learning to Generate Parameters of ConvNets for Unseen Image Datacited by 0
  38. 2024LDCA: Local Descriptors with Contextual Augmentation for Few-Shot Learningcited by 0

50 most cited of 55 All 55 (JSON)