Data to 14 Oct 2026

FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition

arXiv 2312.07536 ↗First posted 12 Dec 2023cs.CVcited by 91

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent approaches such as ControlNet offer users fine-grained spatial control over text-to-image (T2I) diffusion models. However, auxiliary modules have to be trained for each type of spatial condition, model architecture, and checkpoint, putting them at odds with the diverse intents and preferences a human designer would like to convey to the AI models during the content creation process. In this work, we present FreeControl, a training-free approach for controllable T2I generation that supports multiple conditions, architectures, and checkpoints simultaneously. FreeControl designs structure guidance to facilitate the structure alignment with a guidance image, and appearance guidance to enable the appearance sharing between images generated using the same seed. Extensive qualitative and quantitative experiments demonstrate the superior performance of FreeControl across a variety of pre-trained T2I models. In particular, FreeControl facilitates convenient training-free control over many different architectures and checkpoints, allows the challenging input conditions on which most of the existing training-free methods fail, and achieves competitive synthesis quality with training-based approaches.

Key facts. Cited by 91 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 57 of its 60 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

57 of 60 references.

Closest ancestors · 20

Same direction, newest first

8 more

Other references · 37

2017Attention Is All You Needcited by 60,361
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
2020Denoising Diffusion Probabilistic Modelscited by 21,076
2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
2020Denoising Diffusion Implicit Modelscited by 9,781
2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974

25 more

2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
2022Classifier-Free Diffusion Guidancecited by 5,847
2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,064
2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
2023Zero-shot Image-to-Image Translationcited by 505
2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
2023Training-Free Layout Control with Cross-Attention Guidancecited by 313
2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 290
2022EDICT: Exact Diffusion Inversion via Coupled Transformationscited by 258
2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 231
2022Splicing ViT Features for Semantic Appearance Transfercited by 209
2023Expressive Text-to-Image Generation with Rich Textcited by 89

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 34; 2025: 39; 2026: 18.

91 papers.

Sort:

By area:

2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
2025EasyControl: Adding Efficient and Flexible Control for Diffusion Transformercited by 119
2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 105
2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 83
2024Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidancecited by 63
2024Video Diffusion Models are Training-free Motion Interpreter and Controllercited by 61
2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Designcited by 54
2024Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidancecited by 34
2024Understanding and Improving Training-free Loss-based Diffusion Guidancecited by 32
2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
2024SmartControl: Enhancing ControlNet for Handling Rough Visual Conditionscited by 25

38 more

2024FreeTuner: Any Subject in Any Style with Training-free Diffusioncited by 24
2024AnyControl: Create Your Artwork with Versatile Control on Text-to-Image Generationcited by 20
2025Training-Free Constrained Generation With Stable Diffusion Modelscited by 20
2024Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjectscited by 16
2024Denoising with a Joint-Embedding Predictive Architecturecited by 14
2025CoDe: Blockwise Control for Denoising Diffusion Modelscited by 14
2025ComfyGPT: A Self-Optimizing Multi-Agent System for Comprehensive ComfyUI Workflow Generationcited by 14
2024PartCraft: Crafting Creative Objects by Partscited by 13
2025DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generationcited by 13
2024SpotActor: Training-Free Layout-Controlled Consistent Image Generationcited by 12
2025AnyI2V: Animating Any Conditional Image with Motion Controlcited by 11
2025UniHOI: Unified Human-Object Interaction Understanding via Unified Token Spacecited by 10
2024Box It to Bind It: Unified Layout Control and Attribute Binding in T2I Diffusion Modelscited by 9
2024Mojito: Motion Trajectory and Intensity Control for Video Generationcited by 9
2025MultiRef: Controllable Image Generation with Multiple Visual Referencescited by 9
2025SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulationcited by 9
2024Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generationcited by 8
2025FreeBlend: Advancing Concept Blending with Staged Feedback-Driven Interpolation Diffusioncited by 8
2025MirrorVerse: Pushing Diffusion Models to Realistically Reflect the Worldcited by 8
2026Re-Align: Structured Reasoning-guided Alignment for In-Context Image Generation and Editingcited by 8
2024ByTheWay: Boost Your Text-to-Video Generation Model to Higher Quality in a Training-free Waycited by 7
2025Training-free Dense-Aligned Diffusion Guidance for Modular Conditional Image Synthesiscited by 6
2025SUB: Benchmarking CBM Generalization via Synthetic Attribute Substitutionscited by 6
2024ZoDi: Zero-Shot Domain Adaptation with Diffusion-Based Image Transfercited by 5
2025Towards Generating Realistic 3D Semantic Training Data for Autonomous Drivingcited by 5
2025STORK: Faster Diffusion And Flow Matching Sampling By Resolving Both Stiffness And Structure-Dependencecited by 5
2025Dreamland: Controllable World Creation with Simulator and Generative Modelscited by 5
2025Rethink Sparse Signals for Pose-guided Text-to-image Generationcited by 5
2025NeuralRemaster: Phase-Preserving Diffusion for Structure-Aligned Generationcited by 5
2024360PanT: Training-Free Text-Driven 360-Degree Panorama-to-Panorama Translationcited by 4
2024GCA-3D: Towards Generalized and Consistent Domain Adaptation of 3D Generatorscited by 4
2026SeeThrough3D: Occlusion Aware 3D Control in Text-to-Image Generationcited by 4
2024TextCenGen: Attention-Guided Text-Centric Background Adaptation for Text-to-Image Generationcited by 3
2024Event-Customized Image Generationcited by 3
2024High-Resolution Image Synthesis via Next-Token Predictioncited by 3
2025UniCon: Unidirectional Information Flow for Effective Control of Large-Scale Diffusion Modelscited by 3
2025Subjective Camera 1.0: Bridging Human Cognition and Visual Reconstruction through Sequence-Aware Sketch-Guided Diffusioncited by 3
2025Stitch: Training-Free Position Control in Multimodal Diffusion Transformerscited by 3

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way