mlsanityData to 7 Oct 2026

Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights

arXiv 2209.14733 ↗First posted 29 Sep 2022cs.LG cs.CVcited by 63

Not on the map. Why?

Abstract (arXiv)

Learning representations of neural network weights given a model zoo is an emerging and challenging area with many potential applications from model inspection, to neural architecture search or knowledge distillation. Recently, an autoencoder trained on a model zoo was able to learn a hyper-representation, which captures intrinsic and extrinsic properties of the models in the zoo. In this work, we extend hyper-representations for generative use to sample new model weights. We propose layer-wise loss normalization which we demonstrate is key to generate high-performing models and several sampling methods based on the topology of hyper-representations. The models generated using our methods are diverse, performant and capable to outperform strong baselines as evaluated on several downstream tasks: initialization, ensemble sampling and transfer learning. Our results indicate the potential of knowledge aggregation from model zoos to new models via hyper-representations thereby paving the avenue for novel research directions.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

35 of 51 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  5. 2014How transferable are features in deep neural networks?cited by 1,216
  6. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  7. 2016HyperNetworkscited by 1,003
  8. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  9. 2019A Closer Look at Few-shot Classificationcited by 679
  10. 2018Rethinking ImageNet Pre-trainingcited by 376
  11. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  12. 2019A Baseline for Few-Shot Image Classificationcited by 298
23 more
  1. 2015Semi-Supervised Learning with Ladder Networkscited by 246
  2. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  3. 2013Predicting Parameters in Deep Learningcited by 223
  4. 2018Graph HyperNetworks for Neural Architecture Searchcited by 131
  5. 2019From Variational to Deterministic Autoencoderscited by 131
  6. 2019Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processescited by 131
  7. 2015All you need is a good initcited by 124
  8. 2021Instance-Conditioned GANcited by 119
  9. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 113
  10. 2020Predicting Neural Network Accuracy from Weightscited by 107
  11. 2016Learning feed-forward one-shot learnerscited by 93
  12. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  13. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  14. 2022HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learningcited by 55
  15. 2021Factors of Influence for Transfer Learning across Diverse Appearance Domains and Task Typescited by 54
  16. 2019HyperGAN: A Generative Model for Diverse, Performant Neural Networkscited by 51
  17. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
  18. 2019Knowledge Flow: Improve Upon Your Teacherscited by 36
  19. 2021Zoo-Tuning: Adaptive Transfer from a Zoo of Modelscited by 35
  20. 2018Adversarial Distillation of Bayesian Neural Network Posteriorscited by 27
  21. 2019Auto-Embedding Generative Adversarial Networks for High Resolution Image Synthesiscited by 11
  22. 2018Generating Neural Networks with Neural Networkscited by 10
  23. 2019An Acceleration Framework for High Resolution Image Synthesiscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 16; 2025: 21; 2026: 21.

63 papers.

Sort:

By area:

  1. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  2. 2024Graph Neural Networks for Learning Equivariant Representations of Neural Networkscited by 59
  3. 2024Towards Scalable and Versatile Weight Space Learningcited by 40
  4. 2024Neural Network Diffusioncited by 39
  5. 2024Diffusion-Based Neural Network Weights Generationcited by 38
  6. 2023Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?cited by 29
  7. 2025We Should Chart an Atlas of All the World's Modelscited by 25
  8. 2024Scale Equivariant Graph Metanetworkscited by 23
  9. 2024HyperFast: Instant Classification for Tabular Datacited by 22
  10. 2023Sampling weights of deep neural networkscited by 20
  11. 2024Conditional LoRA Parameter Generationcited by 16
  12. 2024Monomial Matrix Group Equivariant Neural Functional Networkscited by 15
38 more
  1. 2025Recurrent Diffusion for Large-Scale Parameter Generationcited by 15
  2. 2024Spatio-Temporal Few-Shot Learning via Diffusive Neural Network Generationcited by 14
  3. 2024Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusioncited by 12
  4. 2025Generative Modeling of Weights: Generalization or Memorization?cited by 12
  5. 2024Unsupervised Model Tree Heritage Recoverycited by 11
  6. 2022Meta-Learning via Classifier(-free) Diffusion Guidancecited by 9
  7. 2024Learning on Model Weights using Tree Expertscited by 7
  8. 2025GradMetaNet: An Equivariant Architecture for Learning on Gradientscited by 7
  9. 2025MedFuncta: A Unified Framework for Learning Efficient Medical Neural Fieldscited by 6
  10. 2025ORAL: Prompting Your Large-Scale LoRAs via Conditional Recurrent Diffusioncited by 6
  11. 2025Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstructioncited by 5
  12. 2025Text2Weight: Bridging Natural Language and Neural Network Weight Spacescited by 5
  13. 2023Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Modelscited by 4
  14. 2024Neural Metamorphosiscited by 4
  15. 2025Learning to Learn Weight Generation via Local Consistency Diffusioncited by 4
  16. 2025From Parameters to Behaviors: Unsupervised Compression of the Policy Spacecited by 4
  17. 2025Learning Model Representations Using Publicly Available Model Hubscited by 4
  18. 2025Instruction-Guided Autoregressive Neural Network Parameter Generationcited by 3
  19. 2025A Model Zoo of Vision Transformerscited by 3
  20. 2025A Model Zoo on Phase Transitions in Neural Networkscited by 3
  21. 2026DeepWeightFlow: Re-Basined Flow Matching for Generating Neural Network Weightscited by 3
  22. 2024Unleash Graph Neural Networks from Heavy Tuningcited by 2
  23. 2025Few-shot Implicit Function Generation via Equivariancecited by 2
  24. 2025Deploying Models to Non-participating Clients in Federated Learning without Fine-tuning: A Hypernetwork-based Approachcited by 2
  25. 2026Position: Weight Space Should Be a First-Class Generative AI Modalitycited by 2
  26. 2024Diffusing to the Top: Boost Graph Neural Networks with Minimal Hyperparameter Tuningcited by 1
  27. 2025The Impact of Model Zoo Size and Composition on Weight Space Learningcited by 1
  28. 2025HyperFlow: Gradient-Free Emulation of Few-Shot Fine-Tuningcited by 1
  29. 2025iLTM: Integrated Large Tabular Modelcited by 1
  30. 2026Ensuring Semantics in Weights of Implicit Neural Representations through the Implicit Function Theoremcited by 1
  31. 2026A Graph Meta-Network for Learning on Kolmogorov-Arnold Networkscited by 1
  32. 2026A Unified Framework for Knowledge Transfer in Bidirectional Model Scalingcited by 1
  33. 2026GeoSANE: Learning Geospatial Representations from Models, Not Datacited by 1
  34. 2026Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matchingcited by 1
  35. 2026LLM-HYPER: Generative CTR Modeling for Cold-Start Ad Personalization via LLM-Based Hypernetworkscited by 1
  36. 2026Quasi-Equivariant Metanetworkscited by 1
  37. 2026ModelLens: Finding the Best for Your Task from Myriads of Modelscited by 1
  38. 2026What Linear Probes Miss: Multi-View Probing for Weight-Space Learningcited by 1

50 most cited of 63 All 63 (JSON)