mlsanityData to 7 Oct 2026

T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models

arXiv 2302.08453 ↗First posted 16 Feb 2023cs.CV cs.AI cs.LGcited by 1,339

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

The incredible generative ability of large-scale text-to-image (T2I) models has demonstrated strong power of learning complex structures and meaningful semantics. However, relying solely on text prompts cannot fully take advantage of the knowledge learned by the model, especially when flexible and accurate controlling (e.g., color and structure) is needed. In this paper, we aim to ``dig out" the capabilities that T2I models have implicitly learned, and then explicitly use them to control the generation more granularly. Specifically, we propose to learn simple and lightweight T2I-Adapters to align internal knowledge in T2I models with external control signals, while freezing the original large T2I models. In this way, we can train various adapters according to different conditions, achieving rich control and editing effects in the color and structure of the generation results. Further, the proposed T2I-Adapters have attractive properties of practical value, such as composability and generalization ability. Extensive experiments demonstrate that our T2I-Adapter has promising generation quality and a wide range of applications.

Key facts. Cited by 1,339 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 43 of its 60 references that are in the map. First posted on arXiv 16 Feb 2023.

Builds on

43 of 60 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  2. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  3. 2022Sketch-Guided Text-to-Image Diffusion Models24 Nov
  4. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Models23 Nov
  5. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  6. 2022Best Prompts for Text-to-Image Models and How to Find Them23 Sep
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  8. 2022Pretraining is All You Need for Image-to-Image Translation25 May
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
20 more
  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  4. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  5. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  6. 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Networkcited by 1,199
  7. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  8. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  9. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  10. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  11. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  12. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  13. 2017Generative Adversarial Networks: An Overviewcited by 588
  14. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  15. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  16. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  17. 2021Pixel Difference Networks for Efficient Edge Detectioncited by 167
  18. 2020You Only Need Adversarial Supervision for Semantic Image Synthesiscited by 129
  19. 2021Multimodal Conditional Image Synthesis with Product-of-Experts GANscited by 75
  20. 2022SceneComposer: Any-Level Semantic Image Synthesiscited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 246; 2024: 449; 2025: 399; 2026: 245.

1,339 papers.

Sort:

By area:

  1. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  2. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  3. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  4. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  5. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  6. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  7. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  8. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  9. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  10. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  11. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  12. 2023Exploiting Diffusion Prior for Real-World Image Super-Resolutioncited by 486
38 more
  1. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  2. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  3. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  4. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  5. 2024InstantID: Zero-shot Identity-Preserving Generation in Secondscited by 431
  6. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  7. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  8. 2024OmniGen: Unified Image Generationcited by 388
  9. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Modelscited by 358
  10. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  11. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  12. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 334
  13. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 324
  14. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 324
  15. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  16. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  17. 2023SeeSR: Towards Semantics-Aware Real-World Image Super-Resolutioncited by 311
  18. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 307
  19. 2025MapAnything: Universal Feed-Forward Metric 3D Reconstructioncited by 295
  20. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Modelcited by 290
  21. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  22. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  23. 2024Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wildcited by 281
  24. 2024Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidancecited by 271
  25. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  26. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  27. 2023DiffBIR: Towards Blind Image Restoration with Generative Diffusion Priorcited by 255
  28. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  29. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  30. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 240
  31. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
  32. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  33. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  34. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  35. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 197
  36. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 197
  37. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Modelscited by 193
  38. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 182

50 most cited of 1,339 All 1,339 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
  2. 2025Decoder-Only LLMs are Better Controllers for Diffusion Models
  3. 2023KOALA: Empirical Lessons Toward Memory-Efficient and Fast Diffusion Models for Text-to-Image Synthesis
  4. 2023DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation
  5. 2026i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models
  6. 2026RAISE: Requirement-Adaptive Evolutionary Refinement for Training-Free Text-to-Image Alignment
  7. 2023If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selection
  8. 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models