# Recurrent Diffusion for Large-Scale Parameter Generation

arXiv: https://arxiv.org/abs/2501.11587 · HTML: https://mlsanity.com/paper/2501.11587/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 76 references that are in the map.
- First posted on arXiv 20 Jan 2025.

## Builds on (most cited first, 30 of 54)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)

## Built on by (most cited first, 15 of 15)

- [Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2505.02567/) (2025)
- [SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass](https://mlsanity.com/paper/2602.06358/) (2026)
- [Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights](https://mlsanity.com/paper/2506.16406/) (2025)
- [Generative Modeling of Weights: Generalization or Memorization?](https://mlsanity.com/paper/2506.07998/) (2025)
- [ORAL: Prompting Your Large-Scale LoRAs via Conditional Recurrent Diffusion](https://mlsanity.com/paper/2503.24354/) (2025)
- [Position: Weight Space Should Be a First-Class Generative AI Modality](https://mlsanity.com/paper/2605.18632/) (2026)
- [DeepWeightFlow: Re-Basined Flow Matching for Generating Neural Network Weights](https://mlsanity.com/paper/2601.05052/) (2026)
- [Learning Model Representations Using Publicly Available Model Hubs](https://mlsanity.com/paper/2510.02096/) (2025)
- [A Model Zoo on Phase Transitions in Neural Networks](https://mlsanity.com/paper/2504.18072/) (2025)
- [HY-WU (Part I): An Extensible Functional Neural Memory Framework and An Instantiation in Text-Guided Image Editing](https://mlsanity.com/paper/2603.07236/) (2026)
- [Prompt2Fingerprint: Plug-and-Play LLM Fingerprinting via Text-to-Weight Generation](https://mlsanity.com/paper/2605.18474/) (2026)
- [Make Optimization Once and for All with Fine-grained Guidance](https://mlsanity.com/paper/2503.11462/) (2025)
- [Flows and Diffusions on the Neural Manifold](https://mlsanity.com/paper/2507.10623/) (2025)
- [Robotic Policy Adaptation via Weight-Space Meta-Learning](https://mlsanity.com/paper/2606.07217/) (2026)
- [SOLAR: A Self-Optimizing Open-Ended Autonomous Agent for Lifelong Learning and Continual Adaptation](https://mlsanity.com/paper/2605.20189/) (2026)

## Abstract (arXiv)

Parameter generation has long struggled to match the scale of today large vision and language models, curbing its broader utility. In this paper, we introduce Recurrent Diffusion for Large Scale Parameter Generation (RPG), a novel framework that generates full neural network parameters up to hundreds of millions on a single GPU. Our approach first partitions a networks parameters into non-overlapping tokens, each corresponding to a distinct portion of the model. A recurrent mechanism then learns the inter token relationships, producing prototypes which serve as conditions for a diffusion process that ultimately synthesizes the full parameters. Across a spectrum of architectures and tasks including ResNets, ConvNeXts and ViTs on ImageNet 1K and COCO, and even LoRA based LLMs RPG achieves performance on par with fully trained networks while avoiding excessive memory overhead. Notably, it generalizes beyond its training set to generate valid parameters for previously unseen tasks, highlighting its flexibility in dynamic and open ended scenarios. By overcoming the longstanding memory and scalability barriers, RPG serves as a critical advance in AI generating AI, potentially enabling efficient weight generation at scales previously deemed infeasible.