mlsanityData to 14 Oct 2026

Hyper-Representations: Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction

arXiv 2110.15288 ↗First posted 28 Oct 2021cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Self-Supervised Learning (SSL) has been shown to learn useful and information-preserving representations. Neural Networks (NNs) are widely applied, yet their weight space is still not fully understood. Therefore, we propose to use SSL to learn hyper-representations of the weights of populations of NNs. To that end, we introduce domain specific data augmentations and an adapted attention architecture. Our empirical evaluation demonstrates that self-supervised representation learning in this domain is able to recover diverse NN model characteristics. Further, we show that the proposed learned representations outperform prior work for predicting hyper-parameters, test accuracy, and generalization gap as well as transfer to out-of-distribution settings.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 36 of its 55 references that are in the map. First posted on arXiv 28 Oct 2021.

Builds on

36 of 55 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  8. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  10. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  12. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
24 more
  1. 2017Random Erasing Data Augmentationcited by 1,239
  2. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
  3. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  4. 2016HyperNetworkscited by 1,027
  5. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  6. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  7. 2015Understanding Neural Networks Through Deep Visualizationcited by 334
  8. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  9. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  10. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
  11. 2014Qualitatively characterizing neural network optimization problemscited by 188
  12. 2015Visualizing and Understanding Recurrent Networkscited by 160
  13. 2020Intriguing Properties of Contrastive Lossescited by 130
  14. 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 113
  15. 2020Predicting Neural Network Accuracy from Weightscited by 110
  16. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  17. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 105
  18. 2020Classifying the classifier: dissecting the weight space of neural networkscited by 65
  19. 2018Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representationcited by 62
  20. 2020Computing the Testing Error without a Testing Setcited by 54
  21. 2019Towards Task and Architecture-Independent Generalization Gap Predictorscited by 18
  22. 2019On Geometric Structure of Activation Spaces in Neural Networkscited by 4
  23. 2019Subspace Match Probably Does Not Accurately Assess the Similarity of Learned Representationscited by 2
  24. 2020An Investigation of the Weight Space to Monitor the Training Progress of Neural Networkscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 1; 2024: 3; 2025: 0; 2026: 2.

9 papers.

  1. 2022From data to functa: Your data point is a function and you can treat it like onecited by 187
  2. 2024Learning Useful Representations of Recurrent Neural Network Weight Matricescited by 15
  3. 2022NeRN -- Learning Neural Representations for Neural Networkscited by 14
  4. 2022Hyper-Representations for Pre-Training and Transfer Learningcited by 12
  5. 2024Implicit-Zoo: A Large-Scale Dataset of Neural Implicit Functions for 2D Images and 3D Scenescited by 12
  6. 2024Meta-Models: An Architecture for Decoding LLM Behaviors Through Interpreted Embeddings and Natural Languagecited by 5
  7. 2026A Graph Meta-Network for Learning on Kolmogorov-Arnold Networkscited by 2
  8. 2026GeoSANE: Learning Geospatial Representations from Models, Not Datacited by 2
  9. 2023Can we infer the presence of Differential Privacy in Deep Learning models' weights? Towards more secure Deep Learningcited by 0