# Model Zoos: A Dataset of Diverse Populations of Neural Network Models

arXiv: https://arxiv.org/abs/2209.14764 · HTML: https://mlsanity.com/paper/2209.14764/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 65 references that are in the map.
- First posted on arXiv 29 Sep 2022.

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [Real Time Image Saliency for Black Box Classifiers](https://mlsanity.com/paper/1705.07857/) (2017)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [A System for Massively Parallel Hyperparameter Tuning](https://mlsanity.com/paper/1810.05934/) (2018)
- [Qualitatively characterizing neural network optimization problems](https://mlsanity.com/paper/1412.6544/) (2014)
- [Visualizing Deep Neural Network Decisions: Prediction Difference Analysis](https://mlsanity.com/paper/1702.04595/) (2017)
- [Visualizing and Understanding Recurrent Networks](https://mlsanity.com/paper/1506.02078/) (2015)
- [Graph HyperNetworks for Neural Architecture Search](https://mlsanity.com/paper/1810.05749/) (2018)
- [Predicting the Generalization Gap in Deep Networks with Margin Distributions](https://mlsanity.com/paper/1810.00113/) (2018)
- [Predicting Neural Network Accuracy from Weights](https://mlsanity.com/paper/2002.11448/) (2020)
- [Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training](https://mlsanity.com/paper/2102.02887/) (2021)
- [How to Start Training: The Effect of Initialization and Architecture](https://mlsanity.com/paper/1803.01719/) (2018)
- [Traditional and Heavy-Tailed Self Regularization in Neural Network Models](https://mlsanity.com/paper/1901.08276/) (2019)

## Built on by (most cited first, 30 of 39)

- [Equivariant Architectures for Learning in Deep Weight Spaces](https://mlsanity.com/paper/2301.12780/) (2023)
- [Graph Neural Networks for Learning Equivariant Representations of Neural Networks](https://mlsanity.com/paper/2403.12143/) (2024)
- [Graph Metanetworks for Processing Diverse Neural Architectures](https://mlsanity.com/paper/2312.04501/) (2023)
- [Towards Scalable and Versatile Weight Space Learning](https://mlsanity.com/paper/2406.09997/) (2024)
- [Diffusion-Based Neural Network Weights Generation](https://mlsanity.com/paper/2402.18153/) (2024)
- [We Should Chart an Atlas of All the World's Models](https://mlsanity.com/paper/2503.10633/) (2025)
- [Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How](https://mlsanity.com/paper/2306.03828/) (2023)
- [Monomial Matrix Group Equivariant Neural Functional Networks](https://mlsanity.com/paper/2409.11697/) (2024)
- [Deep Linear Probe Generators for Weight Space Learning](https://mlsanity.com/paper/2410.10811/) (2024)
- [Learning Useful Representations of Recurrent Neural Network Weight Matrices](https://mlsanity.com/paper/2403.11998/) (2024)
- [Recurrent Diffusion for Large-Scale Parameter Generation](https://mlsanity.com/paper/2501.11587/) (2025)
- [Universal dimensions of visual representation](https://mlsanity.com/paper/2408.12804/) (2024)
- [Set-based Neural Network Encoding Without Weight Tying](https://mlsanity.com/paper/2305.16625/) (2023)
- [Learning on Model Weights using Tree Experts](https://mlsanity.com/paper/2410.13569/) (2024)
- [MedFuncta: A Unified Framework for Learning Efficient Medical Neural Fields](https://mlsanity.com/paper/2502.14401/) (2025)
- [Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstruction](https://mlsanity.com/paper/2503.17138/) (2025)
- [Improving Hyperparameter Optimization with Checkpointed Model Weights](https://mlsanity.com/paper/2406.18630/) (2024)
- [Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Models](https://mlsanity.com/paper/2304.13718/) (2023)
- [SecurityNet: Assessing Machine Learning Vulnerabilities on Public Models](https://mlsanity.com/paper/2310.12665/) (2023)
- [Learning Model Representations Using Publicly Available Model Hubs](https://mlsanity.com/paper/2510.02096/) (2025)
- [Flow to Learn: Flow Matching on Neural Network Parameters](https://mlsanity.com/paper/2503.19371/) (2025)
- [Instruction-Guided Autoregressive Neural Network Parameter Generation](https://mlsanity.com/paper/2504.02012/) (2025)
- [A Model Zoo of Vision Transformers](https://mlsanity.com/paper/2504.10231/) (2025)
- [A Model Zoo on Phase Transitions in Neural Networks](https://mlsanity.com/paper/2504.18072/) (2025)
- [Data Augmentations in Deep Weight Spaces](https://mlsanity.com/paper/2311.08851/) (2023)
- [A Graph Meta-Network for Learning on Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2602.16316/) (2026)
- [GeoSANE: Learning Geospatial Representations from Models, Not Data](https://mlsanity.com/paper/2603.23408/) (2026)
- [The Impact of Model Zoo Size and Composition on Weight Space Learning](https://mlsanity.com/paper/2504.10141/) (2025)
- [Flows and Diffusions on the Neural Manifold](https://mlsanity.com/paper/2507.10623/) (2025)
- [Attesting Model Lineage by Consisted Knowledge Evolution with Fine-Tuning Trajectory](https://mlsanity.com/paper/2601.11683/) (2026)

## Abstract (arXiv)

In the last years, neural networks (NN) have evolved from laboratory environments to the state-of-the-art for many real-world problems. It was shown that NN models (i.e., their weights and biases) evolve on unique trajectories in weight space during training. Following, a population of such neural network models (referred to as model zoo) would form structures in weight space. We think that the geometry, curvature and smoothness of these structures contain information about the state of training and can reveal latent properties of individual models. With such model zoos, one could investigate novel approaches for (i) model analysis, (ii) discover unknown learning dynamics, (iii) learn rich representations of such populations, or (iv) exploit the model zoos for generative modelling of NN weights and biases. Unfortunately, the lack of standardized model zoos and available benchmarks significantly increases the friction for further research about populations of NNs. With this work, we publish a novel dataset of model zoos containing systematically generated and diverse populations of NN models for further research. In total the proposed model zoo dataset is based on eight image datasets, consists of 27 model zoos trained with varying hyperparameter combinations and includes 50'360 unique NN models as well as their sparsified twins, resulting in over 3'844'360 collected model states. Additionally, to the model zoo data we provide an in-depth analysis of the zoos and provide benchmarks for multiple downstream tasks. The dataset can be found at www.modelzoos.cc.