# Model Diffusion for Certifiable Few-shot Transfer Learning

arXiv: https://arxiv.org/abs/2502.06970 · HTML: https://mlsanity.com/paper/2502.06970/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data](https://mlsanity.com/paper/1703.11008/) (2017)
- [LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition](https://mlsanity.com/paper/2307.13269/) (2023)
- [Fine-Tuning Language Models with Just Forward Passes](https://mlsanity.com/paper/2305.17333/) (2023)
- [Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples](https://mlsanity.com/paper/1903.03096/) (2019)
- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [Beyond Memorization: Violating Privacy Via Inference with Large Language Models](https://mlsanity.com/paper/2310.07298/) (2023)
- [Tighter risk certificates for neural networks](https://mlsanity.com/paper/2007.12911/) (2020)
- [LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters](https://mlsanity.com/paper/2405.17604/) (2024)
- [Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts](https://mlsanity.com/paper/2406.10471/) (2024)
- [learn2learn: A Library for Meta-Learning Research](https://mlsanity.com/paper/2008.12284/) (2020)
- [Non-Vacuous Generalization Bounds for Large Language Models](https://mlsanity.com/paper/2312.17173/) (2023)
- [Neural Network Diffusion](https://mlsanity.com/paper/2402.13144/) (2024)
- [MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learning](https://mlsanity.com/paper/2307.16424/) (2023)
- [Diffusion-Based Neural Network Weights Generation](https://mlsanity.com/paper/2402.18153/) (2024)
- [ProtoDiff: Learning to Learn Prototypical Networks by Task-Guided Diffusion](https://mlsanity.com/paper/2306.14770/) (2023)
- [More Flexible PAC-Bayesian Meta-Learning by Learning Learning Algorithms](https://mlsanity.com/paper/2402.04054/) (2024)

## Abstract (arXiv)

In contemporary deep learning, a prevalent and effective workflow for solving low-data problems is adapting powerful pre-trained foundation models (FMs) to new tasks via parameter-efficient fine-tuning (PEFT). However, while empirically effective, the resulting solutions lack generalisation guarantees to certify their accuracy - which may be required for ethical or legal reasons prior to deployment in high-importance applications. In this paper we develop a novel transfer learning approach that is designed to facilitate non-vacuous learning theoretic generalisation guarantees for downstream tasks, even in the low-shot regime. Specifically, we first use upstream tasks to train a distribution over PEFT parameters. We then learn the downstream task by a sample-and-evaluate procedure -- sampling plausible PEFTs from the trained diffusion model and selecting the one with the highest likelihood on the downstream data. Crucially, this confines our model hypothesis to a finite set of PEFT samples. In contrast to the typical continuous hypothesis spaces of neural network weights, this facilitates tighter risk certificates. We instantiate our bound and show non-trivial generalization guarantees compared to existing learning approaches which lead to vacuous bounds in the low-shot regime.