# Diffusion-Based Neural Network Weights Generation

arXiv: https://arxiv.org/abs/2402.18153 · HTML: https://mlsanity.com/paper/2402.18153/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 53 references that are in the map.
- First posted on arXiv 28 Feb 2024.

## Builds on (most cited first, 30 of 38)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Meta-Learning with Implicit Gradients](https://mlsanity.com/paper/1909.04630/) (2019)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [Meta-Learning with Latent Embedding Optimization](https://mlsanity.com/paper/1807.05960/) (2018)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.10512/) (2023)
- [MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer](https://mlsanity.com/paper/2303.14389/) (2023)
- [DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation](https://mlsanity.com/paper/2210.07558/) (2022)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning](https://mlsanity.com/paper/2003.04390/) (2020)
- [Graph HyperNetworks for Neural Architecture Search](https://mlsanity.com/paper/1810.05749/) (2018)
- [A Brief Review of Hypernetworks in Deep Learning](https://mlsanity.com/paper/2306.06955/) (2023)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [Meta-Learning with Adaptive Hyperparameters](https://mlsanity.com/paper/2011.00209/) (2020)
- [Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights](https://mlsanity.com/paper/2209.14733/) (2022)
- [Conditional Text Image Generation with Diffusion Models](https://mlsanity.com/paper/2306.10804/) (2023)
- [HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learning](https://mlsanity.com/paper/2201.04182/) (2022)
- [HyperGAN: A Generative Model for Diverse, Performant Neural Networks](https://mlsanity.com/paper/1901.11058/) (2019)
- [Shallow Bayesian Meta Learning for Real-World Few-Shot Recognition](https://mlsanity.com/paper/2101.02833/) (2021)
- [MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learning](https://mlsanity.com/paper/2307.16424/) (2023)

## Built on by (most cited first, 30 of 39)

- [Conditional LoRA Parameter Generation](https://mlsanity.com/paper/2408.01415/) (2024)
- [Recurrent Diffusion for Large-Scale Parameter Generation](https://mlsanity.com/paper/2501.11587/) (2025)
- [Generative Modeling of Weights: Generalization or Memorization?](https://mlsanity.com/paper/2506.07998/) (2025)
- [Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstruction](https://mlsanity.com/paper/2503.17138/) (2025)
- [ORAL: Prompting Your Large-Scale LoRAs via Conditional Recurrent Diffusion](https://mlsanity.com/paper/2503.24354/) (2025)
- [Position: Weight Space Should Be a First-Class Generative AI Modality](https://mlsanity.com/paper/2605.18632/) (2026)
- [Text2Weight: Bridging Natural Language and Neural Network Weight Spaces](https://mlsanity.com/paper/2508.13633/) (2025)
- [Learning an Efficient Optimizer via Hybrid-Policy Sub-Trajectory Balance](https://mlsanity.com/paper/2511.00543/) (2025)
- [DeepWeightFlow: Re-Basined Flow Matching for Generating Neural Network Weights](https://mlsanity.com/paper/2601.05052/) (2026)
- [Learning to Learn Weight Generation via Local Consistency Diffusion](https://mlsanity.com/paper/2502.01117/) (2025)
- [Learning Model Representations Using Publicly Available Model Hubs](https://mlsanity.com/paper/2510.02096/) (2025)
- [ReALLM: A general framework for LLM compression and fine-tuning](https://mlsanity.com/paper/2405.13155/) (2024)
- [Flow to Learn: Flow Matching on Neural Network Parameters](https://mlsanity.com/paper/2503.19371/) (2025)
- [A Model Zoo on Phase Transitions in Neural Networks](https://mlsanity.com/paper/2504.18072/) (2025)
- [HY-WU (Part I): An Extensible Functional Neural Memory Framework and An Instantiation in Text-Guided Image Editing](https://mlsanity.com/paper/2603.07236/) (2026)
- [Unleash Graph Neural Networks from Heavy Tuning](https://mlsanity.com/paper/2405.12521/) (2024)
- [Enhancing Accuracy and Parameter-Efficiency of Neural Representations for Network Parameterization](https://mlsanity.com/paper/2407.00356/) (2024)
- [A Unified Framework for Knowledge Transfer in Bidirectional Model Scaling](https://mlsanity.com/paper/2603.07506/) (2026)
- [GeoSANE: Learning Geospatial Representations from Models, Not Data](https://mlsanity.com/paper/2603.23408/) (2026)
- [Diffusing to the Top: Boost Graph Neural Networks with Minimal Hyperparameter Tuning](https://mlsanity.com/paper/2410.05697/) (2024)
- [The Impact of Model Zoo Size and Composition on Weight Space Learning](https://mlsanity.com/paper/2504.10141/) (2025)
- [Flows and Diffusions on the Neural Manifold](https://mlsanity.com/paper/2507.10623/) (2025)
- [Hyper Diffusion Avatars: Dynamic Human Avatar Generation using Network Weight Space Diffusion](https://mlsanity.com/paper/2509.04145/) (2025)
- [Semantic-guided LoRA Parameters Generation](https://mlsanity.com/paper/2509.10535/) (2025)
- [LLM-HYPER: Generative CTR Modeling for Cold-Start Ad Personalization via LLM-Based Hypernetworks](https://mlsanity.com/paper/2604.12096/) (2026)
- [Robotic Policy Adaptation via Weight-Space Meta-Learning](https://mlsanity.com/paper/2606.07217/) (2026)
- [Puppet-CNN: Continuous Parameter Dynamics for Input-Adaptive Convolutional Networks](https://mlsanity.com/paper/2411.12876/) (2024)
- [Model Diffusion for Certifiable Few-shot Transfer Learning](https://mlsanity.com/paper/2502.06970/) (2025)
- [Reimagining Parameter Space Exploration with Diffusion Models](https://mlsanity.com/paper/2506.17807/) (2025)
- [SBS: Enhancing Parameter-Efficiency of Neural Representations for Neural Networks via Spectral Bias Suppression](https://mlsanity.com/paper/2509.07373/) (2025)

## Abstract (arXiv)

Transfer learning has gained significant attention in recent deep learning research due to its ability to accelerate convergence and enhance performance on new tasks. However, its success is often contingent on the similarity between source and target data, and training on numerous datasets can be costly, leading to blind selection of pretrained models with limited insight into their effectiveness. To address these challenges, we introduce D2NWG, a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning, conditioned on the target dataset. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation, learning the weight distributions of models pretrained on various datasets. This allows for automatic generation of weights that generalize well across both seen and unseen tasks, outperforming state-of-the-art meta-learning methods and pretrained models. Moreover, our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques that rely on task-specific model collections or access to original training data. By modeling the parameter distribution of LLMs, D2NWG enables task-specific parameter generation without requiring additional fine-tuning or large collections of model variants. Extensive experiments show that our method consistently enhances the performance of diverse base models, regardless of their size or complexity, positioning it as a robust solution for scalable transfer learning.