mlsanityData to 7 Oct 2026

Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?

arXiv 2303.04143 ↗First posted 7 Mar 2023cs.LG cs.AI cs.CVcited by 29

Not on the map. Why?

Abstract (arXiv)

Pretraining a neural network on a large dataset is becoming a cornerstone in machine learning that is within the reach of only a few communities with large-resources. We aim at an ambitious goal of democratizing pretraining. Towards that goal, we train and release a single neural network that can predict high quality ImageNet parameters of other neural networks. By using predicted parameters for initialization we are able to boost training of diverse ImageNet models available in PyTorch. When transferred to other datasets, models initialized with predicted parameters also converge faster and reach competitive final performance.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 41 of its 50 references that are in the map. First posted on arXiv 7 Mar 2023.

Builds on

41 of 50 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  9. 2022A ConvNet for the 2020scited by 3,545
  10. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2021Scaling Vision Transformerscited by 993
29 more
  1. 2015Gated Graph Sequence Neural Networkscited by 792
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  3. 2020A Generalization of Transformer Networks to Graphscited by 653
  4. 2018Do Better ImageNet Models Transfer Better?cited by 560
  5. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  6. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  7. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  8. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  9. 2017SMASH: One-Shot Model Architecture Search through HyperNetworkscited by 200
  10. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  11. 2021Zero-Cost Proxies for Lightweight NAScited by 194
  12. 2022Structure-Aware Transformer for Graph Representation Learningcited by 191
  13. 2020The Computational Limits of Deep Learningcited by 188
  14. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  15. 2018Graph HyperNetworks for Neural Architecture Searchcited by 131
  16. 2019Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processescited by 131
  17. 2016Group Sparse Regularization for Deep Neural Networkscited by 89
  18. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  19. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  20. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 64
  21. 2022Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weightscited by 63
  22. 2022HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learningcited by 55
  23. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
  24. 2022NeRN -- Learning Neural Representations for Neural Networkscited by 14
  25. 2022Towards Theoretically Inspired Neural Initialization Optimizationcited by 11
  26. 2020CATCH: Context-based Meta Reinforcement Learning for Transferrable Architecture Searchcited by 10
  27. 2022Breaking the Architecture Barrier: A Method for Efficient Knowledge Transfer Across Networkscited by 4
  28. 2022One Hyper-Initializer for All Network Architectures in Medical Image Analysiscited by 2
  29. 2022Pretraining a Neural Network before Knowing Its Architecturecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 13; 2026: 5.

29 papers.

Sort:

By area:

  1. 2024Towards Scalable and Versatile Weight Space Learningcited by 40
  2. 2024Neural Network Diffusioncited by 39
  3. 2024Diffusion-Based Neural Network Weights Generationcited by 38
  4. 2025Symmetry in Neural Network Parameter Spacescited by 31
  5. 2024Scale Equivariant Graph Metanetworkscited by 23
  6. 2024WAVE: Weight Templates for Adaptive Initialization of Variable-sized Modelscited by 16
  7. 2025Generative Modeling of Weights: Generalization or Memorization?cited by 12
  8. 2024Transferring Core Knowledge via Learngenescited by 10
  9. 2024FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Modelscited by 5
  10. 2025Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstructioncited by 5
  11. 2025Learning an Efficient Optimizer via Hybrid-Policy Sub-Trajectory Balancecited by 5
  12. 2025Learning to Learn Weight Generation via Local Consistency Diffusioncited by 4
17 more
  1. 2025Learning Model Representations Using Publicly Available Model Hubscited by 4
  2. 2025A Model Zoo on Phase Transitions in Neural Networkscited by 3
  3. 2023Generating Interpretable Networks using Hypernetworkscited by 2
  4. 2024Enhancing Accuracy and Parameter-Efficiency of Neural Representations for Network Parameterizationcited by 2
  5. 2026Position: Weight Space Should Be a First-Class Generative AI Modalitycited by 2
  6. 2025The Impact of Model Zoo Size and Composition on Weight Space Learningcited by 1
  7. 2025Starting Positions Matter: A Study on Better Weight Initialization for Neural Network Quantizationcited by 1
  8. 2025Flows and Diffusions on the Neural Manifoldcited by 1
  9. 2025Semantic-guided LoRA Parameters Generationcited by 1
  10. 2026LLM-HYPER: Generative CTR Modeling for Cold-Start Ad Personalization via LLM-Based Hypernetworkscited by 1
  11. 2024LoGAH: Predicting 774-Million-Parameter Transformers using Graph HyperNetworks with 1/100 Parameterscited by 0
  12. 2024Hyper-Representations: Learning from Populations of Neural Networkscited by 0
  13. 2025(Almost) Free Modality Stitching of Foundation Modelscited by 0
  14. 2025Learning from Oblivion: Predicting Knowledge Overflowed Weights via Retrodiction of Forgettingcited by 0
  15. 2026Self-Supervised Weight Templates for Scalable Vision Model Initializationcited by 0
  16. 2026Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transformcited by 0
  17. 2026WeightCLIP: Aligning Datasets and Models for Weight Space Learningcited by 0