mlsanityData to 7 Oct 2026

RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control

arXiv 2405.17401 ↗First posted 27 May 2024cs.LG cs.CV stat.MLcited by 46

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

We propose Reference-Based Modulation (RB-Modulation), a new plug-and-play solution for training-free personalization of diffusion models. Existing training-free approaches exhibit difficulties in (a) style extraction from reference images in the absence of additional style or content text descriptions, (b) unwanted content leakage from reference style images, and (c) effective composition of style and content. RB-Modulation is built on a novel stochastic optimal controller where a style descriptor encodes the desired attributes through a terminal cost. The resulting drift not only overcomes the difficulties above, but also ensures high fidelity to the reference style and adheres to the given text prompt. We also introduce a cross-attention-based feature aggregation scheme that allows RB-Modulation to decouple content and style from the reference image. With theoretical justification and empirical evidence, our framework demonstrates precise extraction and control of content and style in a training-free manner. Further, our method allows a seamless composition of content and style, which marks a departure from the dependency on external adapters or ControlNets.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 46 of its 56 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

46 of 56 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Measuring Style Similarity in Diffusion Models1 Apr
  2. 2024Visual Style Prompting with Swapping Self-Attention20 Feb
  3. 2024InstantID: Zero-shot Identity-Preserving Generation in Seconds15 Jan
  4. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models13 Aug
  5. 2023Wuerstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models1 Jun
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  7. 2022Multi-Concept Customization of Text-to-Image Diffusion8 Dec
  8. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2022Flow Matching for Generative Modelingcited by 5,426
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  12. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
23 more
  1. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  2. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  3. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  4. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  5. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 526
  6. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
  7. 2023HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Modelscited by 224
  8. 2023Style Aligned Image Generation via Shared Attentioncited by 214
  9. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 210
  10. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 197
  11. 2023Key-Locked Rank One Editing for Text-to-Image Personalizationcited by 194
  12. 2023ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAscited by 194
  13. 2024InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generationcited by 166
  14. 2023Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restorationcited by 164
  15. 2023Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Modelscited by 156
  16. 2023Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusioncited by 78
  17. 2023RealFill: Reference-Driven Generation for Authentic Image Completioncited by 58
  18. 2024ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preservingcited by 42
  19. 2023A Theoretical Justification for Image Inpainting using Denoising Diffusion Probabilistic Modelscited by 37
  20. 2022Stochastic Optimal Control for Collective Variable Free Sampling of Molecular Transition Pathscited by 36
  21. 2017Deep Relaxation: partial differential equations for optimizing deep neural networkscited by 27
  22. 2023Generative Modeling with Phase Stochastic Bridgescited by 24
  23. 2020Lecture Notes on Control System Theory and Designcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 25; 2026: 7.

46 papers.

Sort:

By area:

  1. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 222
  2. 2024Semantic Image Inversion and Editing using Rectified Stochastic Differential Equationscited by 162
  3. 2024InstantStyle-Plus: Style Transfer with Content-Preserving in Text-to-Image Generationcited by 51
  4. 2024Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generatorcited by 48
  5. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  6. 2025Attention Distillation: A Unified Approach to Visual Characteristics Transfercited by 32
  7. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 31
  8. 2024UnZipLoRA: Separating Content and Style from a Single Imagecited by 24
  9. 2024Edify Image: High-Quality Image Generation with Pixel Space Laplacian Diffusion Modelscited by 22
  10. 2025Nested Attention: Semantic-aware Attention Values for Concept Personalizationcited by 21
  11. 2025Personalized Image Generation with Deep Generative Models: A Decade Surveycited by 20
  12. 2024Ensemble Kalman Diffusion Guidance: A Derivative-free Method for Inverse Problemscited by 18
34 more
  1. 2025UniDB: A Unified Diffusion Bridge Framework via Stochastic Optimal Controlcited by 18
  2. 2024Magic Insert: Style-Aware Drag-and-Dropcited by 17
  3. 2024X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Modelscited by 15
  4. 2025Color Conditional Generation with Sliced Wasserstein Guidancecited by 10
  5. 2025MARBLE: Material Recomposition and Blending in CLIP-Spacecited by 9
  6. 2024Style3D: Attention-guided Multi-view Style Transfer for 3D Object Generationcited by 8
  7. 2024Conditional Balance: Improving Multi-Conditioning Trade-Offs in Image Generationcited by 8
  8. 2024Constrained Posterior Sampling: Time Series Generation with Hard Constraintscited by 6
  9. 2025SITA: Structurally Imperceptible and Transferable Adversarial Attacks for Stylized Image Generationcited by 6
  10. 2025Training-Free Multi-Style Fusion Through Reference-Based Adaptive Modulationcited by 6
  11. 2024Style-Friendly SNR Sampler for Style-Driven Generationcited by 5
  12. 2025Prompt-Guided Dual Latent Steering for Inversion Problemscited by 5
  13. 2026How to Guide Your Flow: Few-Step Alignment via Flow Map Reward Guidancecited by 5
  14. 2024MUMU: Bootstrapping Multimodal Image Generation from Text-to-Image Datacited by 4
  15. 2025Transformed Low-rank Adaptation via Tensor Decomposition and Its Applications to Text-to-image Modelscited by 3
  16. 2025Diffusion Bridge or Flow Matching? A Unifying Framework and Comparative Analysiscited by 3
  17. 2026Tuning-free Visual Effect Transfer across Videoscited by 3
  18. 2025SubZero: Composing Subject, Style, and Action via Zero-Shot Personalizationcited by 2
  19. 2025Training-Free Reward-Guided Image Editing via Trajectory Optimal Controlcited by 2
  20. 2026OrthoFuse: Training-free Riemannian Fusion of Orthogonal Style-Concept Adapters for Diffusion Modelscited by 2
  21. 2025Your Text Encoder Can Be An Object-Level Watermarking Controllercited by 1
  22. 2025SPIE: Semantic and Structural Post-Training of Image Editing Diffusion Models with AI feedbackcited by 1
  23. 2025A Unified and Fast-Sampling Diffusion Bridge Framework via Stochastic Optimal Controlcited by 1
  24. 2025Stroke2Sketch: Harnessing Stroke Attributes for Training-Free Sketch Generationcited by 1
  25. 2026Solving Inverse Problems with Flow-based Models via Model Predictive Controlcited by 1
  26. 2025Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Modelscited by 0
  27. 2025Single Image Iterative Subject-driven Generation and Editingcited by 0
  28. 2025One-shot Embroidery Customization via Contrastive LoRA Modulationcited by 0
  29. 2025Image Generation Based on Image Style Extractioncited by 0
  30. 2025Text to Sketch Generation with Multi-Stylescited by 0
  31. 2025Training for Identity, Inference for Controllability: A Unified Approach to Tuning-Free Face Personalizationcited by 0
  32. 2026Stochastic Optimal Control Sampling for Diffusion Inverse Problemscited by 0
  33. 2026PoseAlign: Sculpting Pose-Consistent Meshes via Text-Guided Deformationcited by 0
  34. 2026Residual Diffusion Implicit Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026HAM: A Training-Free Style Transfer Approach via Heterogeneous Attention Modulation for Diffusion Models
  2. 2023Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer
  3. 2025Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Drift
  4. 2023ControlStyle: Text-Driven Stylized Image Generation Using Diffusion Priors
  5. 2024Harnessing the Latent Diffusion Model for Training-Free Image Style Transfer
  6. 2024ArtWeaver: Advanced Dynamic Style Integration via Diffusion Model
  7. 2024DiffuseST: Unleashing the Capability of the Diffusion Model for Style Transfer
  8. 2026TRACE: Time-Adaptive Residual Attention Control with Content-Style Decomposition for Training-Free Diffusion Style Transfer