# Escaping Iterative Parameter-Space Noise: Differentially Private Learning with a Hypernetwork

arXiv: https://arxiv.org/abs/2606.26772 · HTML: https://mlsanity.com/paper/2606.26772/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 34 references that are in the map.
- First posted on arXiv 25 Jun 2026.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data](https://mlsanity.com/paper/1610.05755/) (2016)
- [Scalable Private Learning with PATE](https://mlsanity.com/paper/1802.08908/) (2018)
- [Differentially Private Learning Needs Better Features (or Much More Data)](https://mlsanity.com/paper/2011.11660/) (2020)
- [HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models](https://mlsanity.com/paper/2307.06949/) (2023)
- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification](https://mlsanity.com/paper/2007.03813/) (2020)
- [Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning](https://mlsanity.com/paper/2102.12677/) (2021)
- [Learning to Learn with Generative Models of Neural Network Checkpoints](https://mlsanity.com/paper/2209.12892/) (2022)
- [Membership Inference Attacks Against Text-to-image Generation Models](https://mlsanity.com/paper/2210.00968/) (2022)
- [Public Data-Assisted Mirror Descent for Private Model Training](https://mlsanity.com/paper/2112.00193/) (2021)
- [Neural Network Diffusion](https://mlsanity.com/paper/2402.13144/) (2024)
- [Diffusion-Based Neural Network Weights Generation](https://mlsanity.com/paper/2402.18153/) (2024)
- [Private Adaptive Optimization with Side Information](https://mlsanity.com/paper/2202.05963/) (2022)
- [Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds](https://mlsanity.com/paper/2006.13501/) (2020)
- [Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights](https://mlsanity.com/paper/2506.16406/) (2025)
- [Conditional LoRA Parameter Generation](https://mlsanity.com/paper/2408.01415/) (2024)
- [Improving Differentially Private SGD via Randomly Sparsified Gradients](https://mlsanity.com/paper/2112.00845/) (2021)
- [ORAL: Prompting Your Large-Scale LoRAs via Conditional Recurrent Diffusion](https://mlsanity.com/paper/2503.24354/) (2025)
- [DP-AdamW: Investigating Decoupled Weight Decay and Bias Correction in Private Deep Learning](https://mlsanity.com/paper/2511.07843/) (2025)
- [Noise-Aware Differentially Private Regression via Meta-Learning](https://mlsanity.com/paper/2406.08569/) (2024)
- [Differentially Private Subspace Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2601.11113/) (2026)

## Abstract (arXiv)

Differentially private (DP) training of neural networks is often hindered by the large amount of noise required by gradient-based methods such as DP-SGD, which repeatedly inject high-dimensional noise in parameter space throughout training. In this paper, we propose a new framework for DP learning that avoids iterative optimization in parameter space. Instead of updating the target model using privatized gradients, we employ a hypernetwork trained on public datasets to map a private dataset to the parameters of the target model. Specifically, each example is embedded into a low-dimensional representation, the embeddings are aggregated and perturbed to obtain a DP dataset embedding, and the hypernetwork generates the target model parameters from this noisy embedding. Because privacy noise is injected only once into a low-dimensional dataset representation, our approach can significantly reduce the adverse effect of noise. We theoretically show in a synthetic setting that, under a fixed privacy budget, models produced by our approach achieve higher utility than those trained with DP-SGD. Moreover, we apply our approach to LoRA fine-tuning of diffusion models and show that it achieves lower FID than LoRA models trained with DP-SGD and other public-data-guided methods.