mlsanityData to 7 Oct 2026

Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstruction

arXiv 2503.17138 ↗First posted 21 Mar 2025cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

The weights of neural networks (NNs) have recently gained prominence as a new data modality in machine learning, with applications ranging from accuracy and hyperparameter prediction to representation learning or weight generation. One approach to leverage NN weights involves training autoencoders (AEs), using contrastive and reconstruction losses. This allows such models to be applied to a wide variety of downstream tasks, and they demonstrate strong predictive performance and low reconstruction error. However, despite the low reconstruction error, these AEs reconstruct NN models with deteriorated performance compared to the original ones, limiting their usability with regard to model weight generation. In this paper, we identify a limitation of weight-space AEs, specifically highlighting that a structural loss, that uses the Euclidean distance between original and reconstructed weights, fails to capture some features critical for reconstructing high-performing models. We analyze the addition of a behavioral loss for training AEs in weight space, where we compare the output of the reconstructed model with that of the original one, given some common input. We show a strong synergy between structural and behavioral signals, leading to increased performance in all downstream tasks evaluated, in particular NN weights reconstruction and generation.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 39 of its 49 references that are in the map. First posted on arXiv 21 Mar 2025.

Builds on

39 of 49 references.

References · 39

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  8. 2016HyperNetworkscited by 1,003
  9. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  10. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  11. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  12. 2018Graph HyperNetworks for Neural Architecture Searchcited by 131
27 more
  1. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 113
  2. 2020Predicting Neural Network Accuracy from Weightscited by 107
  3. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  4. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  5. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  6. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  7. 2023Permutation Equivariant Neural Functionalscited by 69
  8. 2022Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weightscited by 63
  9. 2020Classifying the classifier: dissecting the weight space of neural networkscited by 62
  10. 2023Deep Learning on Implicit Neural Representations of Shapescited by 62
  11. 2024Graph Neural Networks for Learning Equivariant Representations of Neural Networkscited by 59
  12. 2022HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learningcited by 55
  13. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 55
  14. 2024Learning by Reconstruction Produces Uninformative Features For Perceptioncited by 54
  15. 2019HyperGAN: A Generative Model for Diverse, Performant Neural Networkscited by 51
  16. 2023Neural Functional Transformerscited by 50
  17. 2024Towards Scalable and Versatile Weight Space Learningcited by 40
  18. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 38
  19. 2024Diffusion-Based Neural Network Weights Generationcited by 38
  20. 2023Equivariant Deep Weight Space Alignmentcited by 34
  21. 2023Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?cited by 29
  22. 2024Improved Generalization of Weight Space Networks via Augmentationscited by 24
  23. 2024Conditional LoRA Parameter Generationcited by 16
  24. 2024Learning Useful Representations of Recurrent Neural Network Weight Matricescited by 13
  25. 2022Hyper-Representations for Pre-Training and Transfer Learningcited by 12
  26. 2018Generating Neural Networks with Neural Networkscited by 10
  27. 2018Unifying and Merging Well-trained Deep Neural Networks for Inference Stagecited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Learning Model Representations Using Publicly Available Model Hubscited by 4
  2. 2025A Model Zoo of Vision Transformerscited by 3
  3. 2025A Model Zoo on Phase Transitions in Neural Networkscited by 3
  4. 2025The Impact of Model Zoo Size and Composition on Weight Space Learningcited by 1
  5. 2026Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matchingcited by 1