mlsanityData to 7 Oct 2026

Learning to Learn with Generative Models of Neural Network Checkpoints

arXiv 2209.12892 ↗First posted 26 Sep 2022cs.LG cs.CVcited by 85

Not on the map. Why?

Cited by 22 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We explore a data-driven approach for learning to optimize neural networks. We construct a dataset of neural network checkpoints and train a generative model on the parameters. In particular, our model is a conditional diffusion transformer that, given an initial input parameter vector and a prompted loss, error, or return, predicts the distribution over parameter updates that achieve the desired metric. At test time, it can optimize neural networks with unseen parameters for downstream tasks in just one update. We find that our approach successfully generates parameters for a wide range of loss prompts. Moreover, it can sample multimodal parameter solutions and has favorable scaling properties. We apply our method to different neural network architectures and tasks in supervised and reinforcement learning.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 54 of its 79 references that are in the map. First posted on arXiv 26 Sep 2022.

Builds on

54 of 79 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  8. 2017Proximal Policy Optimization Algorithmscited by 14,441
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
38 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  6. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  11. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  13. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  14. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  15. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  16. 2020Designing Network Design Spacescited by 954
  17. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  18. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  19. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  20. 2016Learning to learn by gradient descent by gradient descentcited by 616
  21. 2020Jukebox: A Generative Model for Musiccited by 535
  22. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
  23. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  24. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  25. 2018Probabilistic Model-Agnostic Meta-Learningcited by 254
  26. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 197
  27. 2016Learning to Optimizecited by 195
  28. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 117
  29. 2017Learned Optimizers that Scale and Generalizecited by 109
  30. 2020On Linear Identifiability of Learned Representationscited by 103
  31. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  32. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  33. 2021Gradients are Not All You Needcited by 82
  34. 2019On Network Design Spaces for Visual Recognitioncited by 68
  35. 2019HyperGAN: A Generative Model for Diverse, Performant Neural Networkscited by 51
  36. 2017Learning Gradient Descent: Better Generalization and Longer Horizonscited by 50
  37. 2017Learning to Optimize Neural Netscited by 41
  38. 2022Practical tradeoffs between memory, compute, and performance in learned optimizerscited by 32

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 9; 2024: 29; 2025: 27; 2026: 15.

85 papers.

Sort:

By area:

  1. 2022Scalable Diffusion Models with Transformerscited by 5,931
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  3. 2023HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusioncited by 152
  4. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  5. 2023Permutation Equivariant Neural Functionalscited by 69
  6. 2022Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weightscited by 63
  7. 2024Graph Neural Networks for Learning Equivariant Representations of Neural Networkscited by 59
  8. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 55
  9. 2023Neural Functional Transformerscited by 50
  10. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 45
  11. 2024Towards Scalable and Versatile Weight Space Learningcited by 40
  12. 2024Neural Network Diffusioncited by 39
38 more
  1. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 38
  2. 2023Geometry-Complete Diffusion for 3D Molecule Generation and Optimizationcited by 36
  3. 2024Interpreting the Weight Space of Customized Diffusion Modelscited by 35
  4. 2023Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?cited by 29
  5. 2024Universal Neural Functionalscited by 25
  6. 2025We Should Chart an Atlas of All the World's Modelscited by 25
  7. 2024Scale Equivariant Graph Metanetworkscited by 23
  8. 2024Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Modelscited by 21
  9. 2023Sampling weights of deep neural networkscited by 20
  10. 2025Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weightscited by 18
  11. 2024Deep Linear Probe Generators for Weight Space Learningcited by 17
  12. 2024Conditional LoRA Parameter Generationcited by 16
  13. 2024Monomial Matrix Group Equivariant Neural Functional Networkscited by 15
  14. 2025Recurrent Diffusion for Large-Scale Parameter Generationcited by 15
  15. 2024Spatio-Temporal Few-Shot Learning via Diffusive Neural Network Generationcited by 14
  16. 2025In-Context Meta LoRA Generationcited by 14
  17. 2024Prioritized Generative Replaycited by 13
  18. 2024Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusioncited by 12
  19. 2025Generative Modeling of Weights: Generalization or Memorization?cited by 12
  20. 2024Unsupervised Model Tree Heritage Recoverycited by 11
  21. 2024DiffLoRA: Generating Personalized Low-Rank Adaptation Weights with Diffusioncited by 11
  22. 2024Text-to-Model: Text-Conditioned Neural Network Diffusion for Train-Once-for-All Personalizationcited by 10
  23. 2026Learning a Generative Meta-Model of LLM Activationscited by 8
  24. 2024Learning on Model Weights using Tree Expertscited by 7
  25. 2025GradMetaNet: An Equivariant Architecture for Learning on Gradientscited by 7
  26. 2024Heuristically Adaptive Diffusion-Model Evolutionary Strategycited by 6
  27. 2025ORAL: Prompting Your Large-Scale LoRAs via Conditional Recurrent Diffusioncited by 6
  28. 2024Improving Hyperparameter Optimization with Checkpointed Model Weightscited by 5
  29. 2025Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstructioncited by 5
  30. 2023Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Modelscited by 4
  31. 2024TrIND: Representing Anatomical Trees by Denoising Diffusion of Implicit Neural Fieldscited by 4
  32. 2024Neural Metamorphosiscited by 4
  33. 2025Celo: Training Versatile Learned Optimizers on a Compute Dietcited by 4
  34. 2025From Parameters to Behaviors: Unsupervised Compression of the Policy Spacecited by 4
  35. 2025Learning Model Representations Using Publicly Available Model Hubscited by 4
  36. 2024WARPD: World model Assisted Reactive Policy Diffusioncited by 3
  37. 2024Learning to Forget using Hypernetworkscited by 3
  38. 2025Instruction-Guided Autoregressive Neural Network Parameter Generationcited by 3

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models
  2. 2026Diffusion Transformers are Provably Optimal In-context Generators
  3. 2024Joint Diffusion models in Continual Learning
  4. 2026Memorisation, convergence and generalisation in generative models
  5. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models
  6. 2024Understanding and Improving Training-free Loss-based Diffusion Guidance
  7. 2025DRDT3: Diffusion-Refined Decision Test-Time Training Model
  8. 2025A solvable model of learning generative diffusion: theory and insights