# Diffusion Meta-Prompting and Steering for Generalizable Foundation Model Adaptation

arXiv: https://arxiv.org/abs/2610.11067 · HTML: https://mlsanity.com/paper/2610.11067/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 11 of its 55 references that are in the map.
- First posted on arXiv 8 Oct 2026.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 11 of 11)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Meta-SGD: Learning to Learn Quickly for Few-Shot Learning](https://mlsanity.com/paper/1707.09835/) (2017)
- [Domain Adaptation via Prompt Learning](https://mlsanity.com/paper/2202.06687/) (2022)
- [Inserting Anybody in Diffusion Models via Celeb Basis](https://mlsanity.com/paper/2306.00926/) (2023)
- [MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learning](https://mlsanity.com/paper/2307.16424/) (2023)
- [SynthID-Image: Image watermarking at internet scale](https://mlsanity.com/paper/2510.09263/) (2025)
- [DPC: Dual-Prompt Collaboration for Tuning Vision-Language Models](https://mlsanity.com/paper/2503.13443/) (2025)
- [Secure and Robust Watermarking for AI-generated Images: A Comprehensive Survey](https://mlsanity.com/paper/2510.02384/) (2025)
- [Prompt Diffusion Robustifies Any-Modality Prompt Learning](https://mlsanity.com/paper/2410.20164/) (2024)

## Abstract (arXiv)

Prompt learning is a popular method for adapting foundation models, but learned prompts are typically task-specific and fail to generalize to new classes, domains, or compositions of tasks. In this paper, we introduce a Diffusion Meta-Prompt (DMP) model , a framework that models the distribution of learned prompts using diffusion models. Given a repository of previously learned prompts, DMP is trained and sampled without access to the original task examples or task losses, and synthesizes new prompts conditioned on natural language task descriptions. To improve the sampling stability, we introduce a test-time steering strategy for DMP, which uses the best training-selected prompt in the repository as a latent anchor during diffusion sampling, without retraining the DMP or accessing test classes. DMP improves generalization across classification, retrieval and text-to-image generation tasks, supports concept composition and negative prompting without explicit training. It reduces storage and inference costs by over 90% compared to prompt retrieval methods. For composite classification, DMP achieves upto 2.0% average gain over prior meta-learning methods across 55 pairs of datasets with gains as high as 8.5% on specific pairs such as Eurosat and Flowers. DMP also enhances cross-task generalization with ~2-9% improvement for hierarchical classification task. We further provide a theoretical guarantee bounding the expected task loss of prompts sampled from a DMP. Code is available: https://github.com/DeepakSridhar/dmp