# Set-based Neural Network Encoding Without Weight Tying

arXiv: https://arxiv.org/abs/2305.16625 · HTML: https://mlsanity.com/paper/2305.16625/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Meta-Learning with Latent Embedding Optimization](https://mlsanity.com/paper/1807.05960/) (2018)
- [Predicting Neural Network Accuracy from Weights](https://mlsanity.com/paper/2002.11448/) (2020)
- [Equivariant Architectures for Learning in Deep Weight Spaces](https://mlsanity.com/paper/2301.12780/) (2023)
- [SetVAE: Learning Hierarchical Composition for Generative Modeling of Set-Structured Data](https://mlsanity.com/paper/2103.15619/) (2021)
- [Permutation Equivariant Neural Functionals](https://mlsanity.com/paper/2302.14040/) (2023)
- [Deep Learning on Implicit Neural Representations of Shapes](https://mlsanity.com/paper/2302.05438/) (2023)
- [Neural Functional Transformers](https://mlsanity.com/paper/2305.13546/) (2023)
- [Model Zoos: A Dataset of Diverse Populations of Neural Network Models](https://mlsanity.com/paper/2209.14764/) (2022)
- [Unsupervised Learning of Equivariant Structure from Sequences](https://mlsanity.com/paper/2210.05972/) (2022)
- [Regularizing Towards Permutation Invariance in Recurrent Models](https://mlsanity.com/paper/2010.13055/) (2020)
- [Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding](https://mlsanity.com/paper/2103.01615/) (2021)
- [Scalable Set Encoding with Universal Mini-Batch Consistency and Unbiased Full Set Gradient Approximation](https://mlsanity.com/paper/2208.12401/) (2022)

## Built on by (most cited first, 9 of 9)

- [Graph Metanetworks for Processing Diverse Neural Architectures](https://mlsanity.com/paper/2312.04501/) (2023)
- [Equivariant Deep Weight Space Alignment](https://mlsanity.com/paper/2310.13397/) (2023)
- [Improved Generalization of Weight Space Networks via Augmentations](https://mlsanity.com/paper/2402.04081/) (2024)
- [Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models](https://mlsanity.com/paper/2410.04207/) (2024)
- [Position: Weight Space Should Be a First-Class Generative AI Modality](https://mlsanity.com/paper/2605.18632/) (2026)
- [Data Augmentations in Deep Weight Spaces](https://mlsanity.com/paper/2311.08851/) (2023)
- [Dynamic Neural Graph Encoding of Inference Processes in Deep Weight Space](https://mlsanity.com/paper/2607.02166/) (2026)
- [Hyper-Representations: Learning from Populations of Neural Networks](https://mlsanity.com/paper/2410.05107/) (2024)
- [Finite Probes Suffice: Identifiability and Universality for Weight-Space Learning](https://mlsanity.com/paper/2609.33901/) (2026)

## Abstract (arXiv)

We propose a neural network weight encoding method for network property prediction that utilizes set-to-set and set-to-vector functions to efficiently encode neural network parameters. Our approach is capable of encoding neural networks in a model zoo of mixed architecture and different parameter sizes as opposed to previous approaches that require custom encoding models for different architectures. Furthermore, our Set-based Neural network Encoder (SNE) takes into consideration the hierarchical computational structure of neural networks. To respect symmetries inherent in network weight space, we utilize Logit Invariance to learn the required minimal invariance properties. Additionally, we introduce a textitpad-chunk-encode pipeline to efficiently encode neural network layers that is adjustable to computational and memory constraints. We also introduce two new tasks for neural network property prediction: cross-dataset and cross-architecture. In cross-dataset property prediction, we evaluate how well property predictors generalize across model zoos trained on different datasets but of the same architecture. In cross-architecture property prediction, we evaluate how well property predictors transfer to model zoos of different architecture not seen during training. We show that SNE outperforms the relevant baselines on standard benchmarks.