# Controllable Diverse Sampling through Pullback Directions

arXiv: https://arxiv.org/abs/2610.14294 · HTML: https://mlsanity.com/paper/2610.14294/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 25 of its 27 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 25 of 25)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [The Vendi Score: A Diversity Evaluation Metric for Machine Learning](https://mlsanity.com/paper/2210.02410/) (2022)
- [LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation](https://mlsanity.com/paper/2303.17189/) (2023)
- [Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control](https://mlsanity.com/paper/2409.08861/) (2024)
- [BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion](https://mlsanity.com/paper/2403.06976/) (2024)
- [CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling](https://mlsanity.com/paper/2310.17347/) (2023)
- [Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometry](https://mlsanity.com/paper/2307.12868/) (2023)
- [ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization](https://mlsanity.com/paper/2406.04312/) (2024)
- [Particle Guidance: non-I.I.D. Diverse Sampling with Diffusion Models](https://mlsanity.com/paper/2310.13102/) (2023)
- [Consistency-diversity-realism Pareto fronts of conditional image generative models](https://mlsanity.com/paper/2406.10429/) (2024)
- [The Crystal Ball Hypothesis in diffusion models: Anticipating object positions from initial noise](https://mlsanity.com/paper/2406.01970/) (2024)
- [Provable Maximum Entropy Manifold Exploration via Diffusion Models](https://mlsanity.com/paper/2506.15385/) (2025)
- [It's Never Too Late: Noise Optimization for Collapse Recovery in Trained Diffusion Models](https://mlsanity.com/paper/2601.00090/) (2025)

## Abstract (arXiv)

Conditional diffusion models can generate high-quality images that align closely with a given prompt. However, samples generated from different random seeds may remain visually similar, limiting creative exploration. Existing approaches can increase variation, but often require additional optimization or sampling overhead, while stronger perturbations may introduce distortions that degrade image quality or prompt alignment. We propose a training-free method that leverages the local pullback geometry induced by the mapping from the conditioning-embedding space to the image space to identify directions of maximal output sensitivity. By steering different sampling trajectories along these directions, we encourage distinct generations while maintaining consistency with the conditioning signal. Experiments on MS-COCO and BrushNet show the efficacy of the method in improving diversity on prompt-conditioned image generation and inpainting.