mlsanityData to 7 Oct 2026

FOCUS: Optimal Control for Multi-Entity World Modeling in Text-to-Image Generation

arXiv 2510.02315 ↗First posted 2 Oct 2025cs.CVcited by 0

On the map: World models → Video-generation world models and simulators

also in 1 more
Abstract (arXiv)

Text-to-image (T2I) models excel on single-entity prompts but struggle with multi-entity scenes, often exhibiting attribute leakage, identity entanglement, and subject omissions. We present a principled theoretical framework that steers sampling toward multi-subject fidelity by casting flow matching (FM) as stochastic optimal control (SOC), yielding a single hyperparameter controlled trade-off between fidelity and object-centric state separation / binding consistency. Within this framework, we derive two architecture-agnostic algorithms: (i) a training-free test-time controller that perturbs the base velocity with a single-pass update, and (ii) Adjoint Matching, a lightweight fine-tuning rule that regresses a control network to a backward adjoint signal. The same formulation unifies prior attention heuristics, extends to diffusion models via a flow--diffusion correspondence, and provides the first fine-tuning route explicitly designed for multi-subject fidelity. In addition, we also introduce FOCUS (Flow Optimal Control for Unentangled Subjects), a probabilistic attention-binding objective compatible with both algorithms. Empirically, on Stable Diffusion 3.5 and FLUX.1, both algorithms consistently improve multi-subject alignment while maintaining base-model style; test-time control runs efficiently on commodity GPUs, and fine-tuned models generalize to unseen prompts.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 33 of its 34 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

33 of 34 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Scalable Diffusion Models with Transformers19 Dec
  2. 2022Flow Matching for Generative Modeling6 Oct
  3. 2021Learning Transferable Visual Models From Natural Language Supervision26 Feb
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  5. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 28

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  9. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  11. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  12. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
16 more
  1. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
  2. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  3. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  4. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  5. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  6. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  7. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  8. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  9. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 222
  10. 2023Divide & Bind Your Attention for Improved Generative Semantic Nursingcited by 80
  11. 2025Adjoint Sampling: Highly Scalable Diffusion Samplers via Adjoint Matchingcited by 78
  12. 2024Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generationcited by 68
  13. 2023CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Modelscited by 54
  14. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervisioncited by 40
  15. 2024Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjectscited by 16
  16. 2025Be Decisive: Noise-Induced Layouts for Multi-Subject Generationcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models
  2. 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models
  3. 2025FocusDPO: Dynamic Preference Optimization for Multi-Subject Personalized Image Generation via Adaptive Focus
  4. 2024Towards Better Text-to-Image Generation Alignment via Attention Modulation
  5. 2026Improving Controllable Generation: Faster Training and Better Performance via x0-Supervision
  6. 2023Local Conditional Controlling for Text-to-Image Diffusion Models
  7. 2024OneActor: Consistent Character Generation via Cluster-Conditioned Guidance
  8. 2024Controllable Generation with Text-to-Image Diffusion Models: A Survey