# Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance

arXiv: https://arxiv.org/abs/2406.07540 · HTML: https://mlsanity.com/paper/2406.07540/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 41 of its 45 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 41)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [Universal Guidance for Diffusion Models](https://mlsanity.com/paper/2302.07121/) (2023)
- [Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16322/) (2023)
- [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023)
- [Diffusion Self-Guidance for Controllable Image Generation](https://mlsanity.com/paper/2306.00986/) (2023)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation](https://mlsanity.com/paper/2303.17189/) (2023)
- [Break-A-Scene: Extracting Multiple Concepts from a Single Image](https://mlsanity.com/paper/2305.16311/) (2023)

## Built on by (most cited first, 30 of 34)

- [OminiControl: Minimal and Universal Control for Diffusion Transformer](https://mlsanity.com/paper/2411.15098/) (2024)
- [InstantStyle-Plus: Style Transfer with Content-Preserving in Text-to-Image Generation](https://mlsanity.com/paper/2407.00788/) (2024)
- [UniCombine: Unified Multi-Conditional Combination with Diffusion Transformer](https://mlsanity.com/paper/2503.09277/) (2025)
- [YoChameleon: Personalized Vision and Language Generation](https://mlsanity.com/paper/2504.20998/) (2025)
- [ConsisLoRA: Enhancing Content and Style Consistency for LoRA-based Style Transfer](https://mlsanity.com/paper/2503.10614/) (2025)
- [BokehDiff: Neural Lens Blur with One-Step Diffusion](https://mlsanity.com/paper/2507.18060/) (2025)
- [DiffArtist: Towards Structure and Appearance Controllable Image Stylization](https://mlsanity.com/paper/2407.15842/) (2024)
- [Cora: Correspondence-aware image editing using few step diffusion](https://mlsanity.com/paper/2505.23907/) (2025)
- [Dreamland: Controllable World Creation with Simulator and Generative Models](https://mlsanity.com/paper/2506.08006/) (2025)
- [Rethink Sparse Signals for Pose-guided Text-to-image Generation](https://mlsanity.com/paper/2506.20983/) (2025)
- [ArtCrafter: Text-Image Aligning Style Transfer via Embedding Reframing](https://mlsanity.com/paper/2501.02064/) (2025)
- [Inverse Virtual Try-On: Generating Multi-Category Product-Style Images from Clothed Individuals](https://mlsanity.com/paper/2505.21062/) (2025)
- [From Geometric Mimicry to Comprehensive Generation: A Context-Informed Multimodal Diffusion Model for Urban Morphology Synthesis](https://mlsanity.com/paper/2409.17049/) (2024)
- [Event-Customized Image Generation](https://mlsanity.com/paper/2410.02483/) (2024)
- [Domain Generalizable Portrait Style Transfer](https://mlsanity.com/paper/2507.04243/) (2025)
- [ShapeWords: Guiding Text-to-Image Synthesis with 3D Shape-Aware Prompts](https://mlsanity.com/paper/2412.02912/) (2024)
- [Hierarchical and Step-Layer-Wise Tuning of Attention Specialty for Multi-Instance Synthesis in Diffusion Transformers](https://mlsanity.com/paper/2504.10148/) (2025)
- [Universal Few-Shot Spatial Control for Diffusion Models](https://mlsanity.com/paper/2509.07530/) (2025)
- [Group Diffusion: Enhancing Image Generation by Unlocking Cross-Sample Collaboration](https://mlsanity.com/paper/2512.10954/) (2025)
- [OcclusionFormer: Arranging Z-Order for Layout-Grounded Image Generation](https://mlsanity.com/paper/2605.21343/) (2026)
- [Appearance Matching Adapter for Exemplar-based Semantic Image Synthesis in-the-Wild](https://mlsanity.com/paper/2412.03150/) (2024)
- [Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Models](https://mlsanity.com/paper/2502.07466/) (2025)
- [Enhancing Product Search Interfaces with Sketch-Guided Diffusion and Language Agents](https://mlsanity.com/paper/2504.08739/) (2025)
- [RichControl: Structure- and Appearance-Rich Training-Free Spatial Control for Text-to-Image Generation](https://mlsanity.com/paper/2507.02792/) (2025)
- [Dual Recursive Feedback on Generation and Appearance Latents for Pose-Robust Text-to-Image Diffusion](https://mlsanity.com/paper/2508.09575/) (2025)
- [TokenAR: Multiple Subject Generation via Autoregressive Token-level enhancement](https://mlsanity.com/paper/2510.16332/) (2025)
- [Self-Attention Decomposition For Training Free Diffusion Editing](https://mlsanity.com/paper/2510.22650/) (2025)
- [Localized Control in Diffusion Models via Latent Vector Prediction](https://mlsanity.com/paper/2602.01991/) (2026)
- [Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scoping](https://mlsanity.com/paper/2602.06850/) (2026)
- [Attention Frequency Modulation: Training-Free Spectral Modulation of Diffusion Cross-Attention](https://mlsanity.com/paper/2603.28114/) (2026)

## Abstract (arXiv)

Recent controllable generation approaches such as FreeControl and Diffusion Self-Guidance bring fine-grained spatial and appearance control to text-to-image (T2I) diffusion models without training auxiliary modules. However, these methods optimize the latent embedding for each type of score function with longer diffusion steps, making the generation process time-consuming and limiting their flexibility and use. This work presents Ctrl-X, a simple framework for T2I diffusion controlling structure and appearance without additional training or guidance. Ctrl-X designs feed-forward structure control to enable the structure alignment with a structure image and semantic-aware appearance transfer to facilitate the appearance transfer from a user-input image. Extensive qualitative and quantitative experiments illustrate the superior performance of Ctrl-X on various condition inputs and model checkpoints. In particular, Ctrl-X supports novel structure and appearance control with arbitrary condition images of any modality, exhibits superior image quality and appearance transfer compared to existing works, and provides instant plug-and-play functionality to any T2I and text-to-video (T2V) diffusion model. See our project page for an overview of the results: https://genforce.github.io/ctrl-x