mlsanityData to 7 Oct 2026

OminiControl: Minimal and Universal Control for Diffusion Transformer

arXiv 2411.15098 ↗First posted 22 Nov 2024cs.CV cs.AI cs.LGcited by 334

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

We present OminiControl, a novel approach that rethinks how image conditions are integrated into Diffusion Transformer (DiT) architectures. Current image conditioning methods either introduce substantial parameter overhead or handle only specific control tasks effectively, limiting their practical versatility. OminiControl addresses these limitations through three key innovations: (1) a minimal architectural design that leverages the DiT's own VAE encoder and transformer blocks, requiring just 0.1% additional parameters; (2) a unified sequence processing strategy that combines condition tokens with image tokens for flexible token interactions; and (3) a dynamic position encoding mechanism that adapts to both spatially-aligned and non-aligned control tasks. Our extensive experiments show that this streamlined approach not only matches but surpasses the performance of specialized methods across multiple conditioning tasks. To overcome data limitations in subject-driven generation, we also introduce Subjects200K, a large-scale dataset of identity-consistent image pairs synthesized using DiT models themselves. This work demonstrates that effective image control can be achieved without architectural complexity, opening new possibilities for efficient and versatile image generation systems.

Key facts. Cited by 334 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 47 of its 62 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

47 of 62 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  2. 2024PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models10 Jan
  3. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis30 Sep
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  5. 2022Scalable Diffusion Models with Transformers19 Dec
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  7. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2022Generative Adversarial Networkscited by 5,603
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  6. 2022Flow Matching for Generative Modelingcited by 5,426
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  9. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  10. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  11. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  12. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
28 more
  1. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  2. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  3. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,058
  4. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
  5. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  6. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  7. 2022MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessmentcited by 514
  8. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  9. 2024InstantID: Zero-shot Identity-Preserving Generation in Secondscited by 431
  10. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 414
  11. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Modelscited by 358
  12. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  13. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 324
  14. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  15. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 197
  16. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  17. 2024Semantic Image Inversion and Editing using Rectified Stochastic Differential Equationscited by 162
  18. 2024In-Context LoRA for Diffusion Transformerscited by 155
  19. 2023SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generationcited by 142
  20. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 129
  21. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  22. 2024ACE: All-round Creator and Editor Following Instructions via Diffusion Transformercited by 56
  23. 2024ControlAR: Controllable Image Generation with Autoregressive Modelscited by 48
  24. 2024Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidancecited by 33
  25. 2020Multi-modal Attention for Speech Emotion Recognitioncited by 29
  26. 2024Group Diffusion Transformers are Unsupervised Multitask Learnerscited by 22
  27. 2024AnyControl: Create Your Artwork with Versatile Control on Text-to-Image Generationcited by 20
  28. 2024Character-Adapter: Prompt-Guided Region Control for High-Fidelity Character Customizationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 198; 2026: 131.

334 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  2. 2025VACE: All-in-One Video Creation and Editingcited by 467
  3. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  4. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  5. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  6. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 225
  7. 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 199
  8. 2025Less-to-More Generalization: Unlocking More Controllability by In-Context Generationcited by 153
  9. 2025EasyControl: Adding Efficient and Flexible Control for Diffusion Transformercited by 118
  10. 2025ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Fillingcited by 95
  11. 2025Controllable Video Generation: A Surveycited by 93
  12. 2025Mixture of Contexts for Long Video Generationcited by 90
38 more
  1. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 85
  2. 2025HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generationcited by 84
  3. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  4. 2025DreamO: A Unified Framework for Image Customizationcited by 82
  5. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  6. 2025Scaling Instruction-Based Video Editing with a High-Quality Synthetic Datasetcited by 76
  7. 2025KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Modelscited by 69
  8. 2025Ovis-U1 Technical Reportcited by 65
  9. 2025HunyuanVideo-Avatar: High-Fidelity Audio-Driven Human Animation for Multiple Characterscited by 64
  10. 2025Any2AnyTryon: Leveraging Adaptive Position Embeddings for Versatile Virtual Clothing Taskscited by 62
  11. 2025Insert Anything: Image Insertion via In-Context Editing in DiTcited by 61
  12. 2025OminiControl2: Efficient Conditioning for Diffusion Transformerscited by 56
  13. 2025XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulationcited by 55
  14. 2025From Reflection to Perfection: Scaling Inference-Time Optimization for Text-to-Image Diffusion Models via Reflection Tuningcited by 53
  15. 2025Inference-time Scaling of Diffusion Models through Classical Searchcited by 48
  16. 2024Diffusion Self-Distillation for Zero-Shot Customized Image Generationcited by 46
  17. 2025MakeAnything: Harnessing Diffusion Transformers for Multi-Domain Procedural Sequence Generationcited by 43
  18. 2024FonTS: Text Rendering with Typography and Style Controlscited by 40
  19. 2025VisualCloze: A Universal Image Generation Framework via Visual In-Context Learningcited by 39
  20. 2025OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpaintingcited by 38
  21. 2025PosterCraft: Rethinking High-Quality Aesthetic Poster Generation in a Unified Frameworkcited by 37
  22. 2025POSTA: A Go-to Framework for Customized Artistic Poster Generationcited by 36
  23. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  24. 2025Lumina-mGPT 2.0: Stand-Alone AutoRegressive Image Modelingcited by 35
  25. 2025DreamOmni2: Multimodal Instruction-based Editing and Generationcited by 35
  26. 2025InfiniteYou: Flexible Photo Recrafting While Preserving Your Identitycited by 34
  27. 2025OmniConsistency: Learning Style-Agnostic Consistency from Paired Stylization Datacited by 32
  28. 2025Omni-Effects: Unified and Spatially-Controllable Visual Effects Generationcited by 27
  29. 2025RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformerscited by 26
  30. 2025USO: Unified Style and Subject-Driven Generation via Disentangled and Reward Learningcited by 26
  31. 2025Generating Multi-Image Synthetic Data for Text-to-Image Customizationcited by 25
  32. 2025UniCombine: Unified Multi-Conditional Combination with Diffusion Transformercited by 25
  33. 2025EasyText: Controllable Diffusion Transformer for Multilingual Text Renderingcited by 25
  34. 2025X2Edit: Revisiting Arbitrary-Instruction Image Editing through Self-Constructed Data and Task-Aware Representation Learningcited by 24
  35. 2025ICE-Bench: A Unified and Comprehensive Benchmark for Image Creating and Editingcited by 23
  36. 2025UniTEX: Universal High Fidelity Generative Texturing for 3D Shapescited by 23
  37. 2025Personalize Anything for Free with Diffusion Transformercited by 22
  38. 2025Region-Constraint In-Context Generation for Instructional Video Editingcited by 22

50 most cited of 334 All 334 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025RAPID^3: Tri-Level Reinforced Acceleration Policies for Diffusion Transformer
  2. 2025OmniCache: A Trajectory-Oriented Global Perspective on Training-Free Cache Reuse for Diffusion Transformer Models
  3. 2024Delta-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers
  4. 2026DiT-IC: Aligned Diffusion Transformer for Efficient Image Compression
  5. 2026Unveiling the Secret of AdaLN-Zero in Diffusion Transformer
  6. 2025Exploring Magnitude Preservation and Rotation Modulation in Diffusion Transformers
  7. 2026TIDE: Text-Informed Dynamic Extrapolation with Step-Aware Temperature Control for Diffusion Transformers
  8. 2024Dynamic Diffusion Transformer