FOCUS: Optimal Control for Multi-Entity World Modeling in Text-to-Image Generation
On the map: World models → Video-generation world models and simulators
also in 1 more
Abstract (arXiv)
Text-to-image (T2I) models excel on single-entity prompts but struggle with multi-entity scenes, often exhibiting attribute leakage, identity entanglement, and subject omissions. We present a principled theoretical framework that steers sampling toward multi-subject fidelity by casting flow matching (FM) as stochastic optimal control (SOC), yielding a single hyperparameter controlled trade-off between fidelity and object-centric state separation / binding consistency. Within this framework, we derive two architecture-agnostic algorithms: (i) a training-free test-time controller that perturbs the base velocity with a single-pass update, and (ii) Adjoint Matching, a lightweight fine-tuning rule that regresses a control network to a backward adjoint signal. The same formulation unifies prior attention heuristics, extends to diffusion models via a flow--diffusion correspondence, and provides the first fine-tuning route explicitly designed for multi-subject fidelity. In addition, we also introduce FOCUS (Flow Optimal Control for Unentangled Subjects), a probabilistic attention-binding objective compatible with both algorithms. Empirically, on Stable Diffusion 3.5 and FLUX.1, both algorithms consistently improve multi-subject alignment while maintaining base-model style; test-time control runs efficiently on commodity GPUs, and fine-tuned models generalize to unseen prompts.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 33 of its 34 references that are in the map. First posted on arXiv 2 Oct 2025.
Builds on
33 of 34 references.
Closest ancestors · 5
Same direction, newest first
- 2022Scalable Diffusion Models with Transformers19 Dec
- 2022Flow Matching for Generative Modeling6 Oct
- 2021Learning Transferable Visual Models From Natural Language Supervision26 Feb
- 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
- 2020Denoising Diffusion Probabilistic Models19 Jun
Other references · 28
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
- 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
- 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
- 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
- 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
- 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
16 more
- 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
- 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
- 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
- 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
- 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
- 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
- 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
- 2023Scaling Open-Vocabulary Object Detectioncited by 335
- 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 222
- 2023Divide & Bind Your Attention for Improved Generative Semantic Nursingcited by 80
- 2025Adjoint Sampling: Highly Scalable Diffusion Samplers via Adjoint Matchingcited by 78
- 2024Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generationcited by 68
- 2023CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Modelscited by 54
- 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervisioncited by 40
- 2024Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjectscited by 16
- 2025Be Decisive: Noise-Induced Layouts for Multi-Subject Generationcited by 10
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models
- 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models
- 2025FocusDPO: Dynamic Preference Optimization for Multi-Subject Personalized Image Generation via Adaptive Focus
- 2024Towards Better Text-to-Image Generation Alignment via Attention Modulation
- 2026Improving Controllable Generation: Faster Training and Better Performance via x0-Supervision
- 2023Local Conditional Controlling for Text-to-Image Diffusion Models
- 2024OneActor: Consistent Character Generation via Cluster-Conditioned Guidance
- 2024Controllable Generation with Text-to-Image Diffusion Models: A Survey