mlsanityData to 7 Oct 2026

Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance

arXiv 2406.07540 ↗First posted 11 Jun 2024cs.CV cs.LGcited by 33

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent controllable generation approaches such as FreeControl and Diffusion Self-Guidance bring fine-grained spatial and appearance control to text-to-image (T2I) diffusion models without training auxiliary modules. However, these methods optimize the latent embedding for each type of score function with longer diffusion steps, making the generation process time-consuming and limiting their flexibility and use. This work presents Ctrl-X, a simple framework for T2I diffusion controlling structure and appearance without additional training or guidance. Ctrl-X designs feed-forward structure control to enable the structure alignment with a structure image and semantic-aware appearance transfer to facilitate the appearance transfer from a user-input image. Extensive qualitative and quantitative experiments illustrate the superior performance of Ctrl-X on various condition inputs and model checkpoints. In particular, Ctrl-X supports novel structure and appearance control with arbitrary condition images of any modality, exhibits superior image quality and appearance transfer compared to existing works, and provides instant plug-and-play functionality to any T2I and text-to-video (T2V) diffusion model. See our project page for an overview of the results: https://genforce.github.io/ctrl-x

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 41 of its 45 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

41 of 45 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition12 Dec
  2. 2023Inversion-Free Image Editing with Natural Language7 Dec
  3. 2023Orthogonal Adaptation for Modular Customization of Diffusion Models5 Dec
  4. 2023R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation13 Oct
  5. 2023Dense Text-to-Image Generation with Attention Modulation24 Aug
  6. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models13 Aug
  7. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models25 May
  8. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation30 Mar
  9. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  11. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  12. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
1 more
  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  12. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
16 more
  1. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  2. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  3. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  4. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  5. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  6. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  7. 2023Universal Guidance for Diffusion Modelscited by 426
  8. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  9. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  10. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 281
  11. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
  12. 2022Splicing ViT Features for Semantic Appearance Transfercited by 209
  13. 2021AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfercited by 178
  14. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  15. 2023Cross-Image Attention for Zero-Shot Appearance Transfercited by 135
  16. 2023RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Modelscited by 113

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 19; 2026: 7.

33 papers.

Sort:

By area:

  1. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 334
  2. 2024InstantStyle-Plus: Style Transfer with Content-Preserving in Text-to-Image Generationcited by 51
  3. 2025UniCombine: Unified Multi-Conditional Combination with Diffusion Transformercited by 25
  4. 2025YoChameleon: Personalized Vision and Language Generationcited by 20
  5. 2025ConsisLoRA: Enhancing Content and Style Consistency for LoRA-based Style Transfercited by 9
  6. 2025BokehDiff: Neural Lens Blur with One-Step Diffusioncited by 8
  7. 2024DiffArtist: Towards Structure and Appearance Controllable Image Stylizationcited by 6
  8. 2025Cora: Correspondence-aware image editing using few step diffusioncited by 5
  9. 2025Dreamland: Controllable World Creation with Simulator and Generative Modelscited by 5
  10. 2025Rethink Sparse Signals for Pose-guided Text-to-image Generationcited by 5
  11. 2025ArtCrafter: Text-Image Aligning Style Transfer via Embedding Reframingcited by 4
  12. 2025Inverse Virtual Try-On: Generating Multi-Category Product-Style Images from Clothed Individualscited by 4
21 more
  1. 2024From Geometric Mimicry to Comprehensive Generation: A Context-Informed Multimodal Diffusion Model for Urban Morphology Synthesiscited by 3
  2. 2024Event-Customized Image Generationcited by 3
  3. 2025Domain Generalizable Portrait Style Transfercited by 3
  4. 2024ShapeWords: Guiding Text-to-Image Synthesis with 3D Shape-Aware Promptscited by 2
  5. 2025Hierarchical and Step-Layer-Wise Tuning of Attention Specialty for Multi-Instance Synthesis in Diffusion Transformerscited by 2
  6. 2025Universal Few-Shot Spatial Control for Diffusion Modelscited by 2
  7. 2025Group Diffusion: Enhancing Image Generation by Unlocking Cross-Sample Collaborationcited by 1
  8. 2026OcclusionFormer: Arranging Z-Order for Layout-Grounded Image Generationcited by 1
  9. 2024Appearance Matching Adapter for Exemplar-based Semantic Image Synthesis in-the-Wildcited by 0
  10. 2025Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Modelscited by 0
  11. 2025Enhancing Product Search Interfaces with Sketch-Guided Diffusion and Language Agentscited by 0
  12. 2025RichControl: Structure- and Appearance-Rich Training-Free Spatial Control for Text-to-Image Generationcited by 0
  13. 2025Dual Recursive Feedback on Generation and Appearance Latents for Pose-Robust Text-to-Image Diffusioncited by 0
  14. 2025TokenAR: Multiple Subject Generation via Autoregressive Token-level enhancementcited by 0
  15. 2025Self-Attention Decomposition For Training Free Diffusion Editingcited by 0
  16. 2026Localized Control in Diffusion Models via Latent Vector Predictioncited by 0
  17. 2026Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scopingcited by 0
  18. 2026Attention Frequency Modulation: Training-Free Spectral Modulation of Diffusion Cross-Attentioncited by 0
  19. 2026Semantic-Structural Alignment for Generative Pictorial Chartscited by 0
  20. 2026AnyStyle: A Single LoRA is Sufficient for Image-Guided Style Transfercited by 0
  21. 2026RankT2I: A Submodular Framework for Discovering Interpretable and Diverse Semantics in Text-to-Image Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Controllable Generation with Text-to-Image Diffusion Models: A Survey
  2. 2024CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation
  3. 2026Improving Controllable Generation: Faster Training and Better Performance via x0-Supervision
  4. 2024FlexEControl: Flexible and Efficient Multimodal Control for Text-to-Image Generation
  5. 2026Layer-wise Instance Binding for Regional and Occlusion Control in Text-to-Image Diffusion Transformers
  6. 2024Box It to Bind It: Unified Layout Control and Attribute Binding in T2I Diffusion Models
  7. 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models
  8. 2025LLMControl: Grounded Control of Text-to-Image Diffusion-based Synthesis with Multimodal LLMs