mlsanityData to 7 Oct 2026

Parameter Prediction for Unseen Deep Architectures

arXiv 2110.13100 ↗First posted 25 Oct 2021cs.LG cs.AI cs.CVcited by 89

Not on the map. Why?

Abstract (arXiv)

Deep learning has been successful in automating the design of features in machine learning pipelines. However, the algorithms optimizing neural network parameters remain largely hand-designed and computationally inefficient. We study if we can use deep learning to directly predict these parameters by exploiting the past knowledge of training other networks. We introduce a large-scale dataset of diverse computational graphs of neural architectures - DeepNets-1M - and use it to explore parameter prediction on CIFAR-10 and ImageNet. By leveraging advances in graph neural networks, we propose a hypernetwork that can predict performant parameters in a single forward pass taking a fraction of a second, even on a CPU. The proposed model achieves surprisingly good performance on unseen and diverse networks. For example, it is able to predict all 24 million parameters of a ResNet-50 achieving a 60% accuracy on CIFAR-10. On ImageNet, top-5 accuracy of some of our networks approaches 50%. Our task along with the model and results can potentially lead to a new, more computationally efficient paradigm of training networks. Our model also learns a strong representation of neural architectures enabling their analysis.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Builds on 97 of its 122 references that are in the map. First posted on arXiv 25 Oct 2021.

Builds on

97 of 122 references.

References · 97

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  8. 2017Graph Attention Networkscited by 8,854
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  11. 2016Densely Connected Convolutional Networkscited by 6,691
  12. 2016Layer Normalizationcited by 5,579
38 more
  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2017Squeeze-and-Excitation Networkscited by 4,763
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2016Wide Residual Networkscited by 3,057
  7. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  10. 2019Searching for MobileNetV3cited by 1,984
  11. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  12. 2018Group Normalizationcited by 1,514
  13. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  14. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  15. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  16. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  17. 2016An overview of gradient descent optimization algorithmscited by 1,021
  18. 2016HyperNetworkscited by 1,003
  19. 2016Deep Networks with Stochastic Depthcited by 973
  20. 2020Designing Network Design Spacescited by 954
  21. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  22. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  23. 2020Geom-GCN: Geometric Graph Convolutional Networkscited by 821
  24. 2015Gated Graph Sequence Neural Networkscited by 792
  25. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  26. 2020Benchmarking Graph Neural Networkscited by 672
  27. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 624
  28. 2016Learning to learn by gradient descent by gradient descentcited by 616
  29. 2018Do Better ImageNet Models Transfer Better?cited by 560
  30. 2017Progressive Neural Architecture Searchcited by 483
  31. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  32. 2019AutoML: A Survey of the State-of-the-Artcited by 388
  33. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 366
  34. 2020What is being transferred in transfer learning?cited by 365
  35. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  36. 2019DAG-GNN: DAG Structure Learning with Graph Neural Networkscited by 329
  37. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  38. 2013Predicting Parameters in Deep Learningcited by 223

50 most cited of 97 All 97 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 21; 2023: 20; 2024: 17; 2025: 21; 2026: 10.

89 papers.

Sort:

By area:

  1. 2022From data to functa: Your data point is a function and you can treat it like onecited by 183
  2. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  3. 2023ATT3D: Amortized Text-to-3D Object Synthesiscited by 87
  4. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  5. 2022Tutorial on amortized optimizationcited by 83
  6. 2023Permutation Equivariant Neural Functionalscited by 69
  7. 2022Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weightscited by 63
  8. 2023Deep Learning on Implicit Neural Representations of Shapescited by 62
  9. 2024Graph Neural Networks for Learning Equivariant Representations of Neural Networkscited by 59
  10. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 55
  11. 2023Neural Functional Transformerscited by 50
  12. 2024Towards Scalable and Versatile Weight Space Learningcited by 40
38 more
  1. 2024Neural Network Diffusioncited by 39
  2. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 38
  3. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  4. 2024Diffusion-Based Neural Network Weights Generationcited by 38
  5. 2023Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?cited by 29
  6. 2022Gradient-based Bi-level Optimization for Deep Learning: A Surveycited by 26
  7. 2022Transformers over Directed Acyclic Graphscited by 23
  8. 2024Scale Equivariant Graph Metanetworkscited by 23
  9. 2023Generating Behaviorally Diverse Policies with Latent Diffusion Modelscited by 18
  10. 2022Federated Learning with Heterogeneous Architectures using Graph HyperNetworkscited by 17
  11. 2024Conditional LoRA Parameter Generationcited by 16
  12. 2024Monomial Matrix Group Equivariant Neural Functional Networkscited by 15
  13. 2022NeRN -- Learning Neural Representations for Neural Networkscited by 14
  14. 2022Hyper-Representations for Pre-Training and Transfer Learningcited by 12
  15. 2025Generative Modeling of Weights: Generalization or Memorization?cited by 12
  16. 2023Amortized Inference for Gaussian Process Hyperparameters of Structured Kernelscited by 10
  17. 2024Transferring Core Knowledge via Learngenescited by 10
  18. 2022Goal-Conditioned Generators of Deep Policiescited by 9
  19. 2023Task-Agnostic Low-Rank Adapters for Unseen English Dialectscited by 9
  20. 2022Teaching Networks to Solve Optimization Problemscited by 7
  21. 2023State of the Art and Potentialities of Graph-level Learningcited by 6
  22. 2023Fast Unsupervised Deep Outlier Model Selection with Hypernetworkscited by 6
  23. 2024Connecting NeRFs, Images, and Textcited by 6
  24. 2024LLaNA: Large Language and NeRF Assistantcited by 6
  25. 2023Deep Learning on Object-centric 3D Neural Fieldscited by 5
  26. 2024FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Modelscited by 5
  27. 2024Architecture-Aware Learning Curve Extrapolation via Graph Ordinary Differential Equationcited by 5
  28. 2025Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstructioncited by 5
  29. 2025Learning an Efficient Optimizer via Hybrid-Policy Sub-Trajectory Balancecited by 5
  30. 2022Efficiently Learning Small Policies for Locomotion and Manipulationcited by 4
  31. 2022Breaking the Architecture Barrier: A Method for Efficient Knowledge Transfer Across Networkscited by 4
  32. 2023HyperPPO: A scalable method for finding small policies for robotic controlcited by 4
  33. 2024Neural Metamorphosiscited by 4
  34. 2025Learning to Learn Weight Generation via Local Consistency Diffusioncited by 4
  35. 2025Learning Model Representations Using Publicly Available Model Hubscited by 4
  36. 2024Younger: The First Dataset for Artificial Intelligence-Generated Neural Network Architecturecited by 3
  37. 2025Instruction-Guided Autoregressive Neural Network Parameter Generationcited by 3
  38. 2025A Model Zoo of Vision Transformerscited by 3

50 most cited of 89 All 89 (JSON)