mlsanityData to 7 Oct 2026

IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models

arXiv 2308.06721 ↗First posted 13 Aug 2023cs.CV cs.AIcited by 1,439

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent years have witnessed the strong power of large text-to-image diffusion models for the impressive generative capability to create high-fidelity images. However, it is very tricky to generate desired images using only text prompt as it often involves complex prompt engineering. An alternative to text prompt is image prompt, as the saying goes: "an image is worth a thousand words". Although existing methods of direct fine-tuning from pretrained models are effective, they require large computing resources and are not compatible with other base models, text prompt, and structural controls. In this paper, we present IP-Adapter, an effective and lightweight adapter to achieve image prompt capability for the pretrained text-to-image diffusion models. The key design of our IP-Adapter is decoupled cross-attention mechanism that separates cross-attention layers for text features and image features. Despite the simplicity of our method, an IP-Adapter with only 22M parameters can achieve comparable or even better performance to a fully fine-tuned image prompt model. As we freeze the pretrained diffusion model, the proposed IP-Adapter can be generalized not only to other custom models fine-tuned from the same base model, but also to controllable generation using existing controllable tools. With the benefit of the decoupled cross-attention strategy, the image prompt can also work well with the text prompt to achieve multimodal image generation. The project page is available at urlhttps://ip-adapter.github.io.

Key facts. Cited by 1,439 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 48 of its 51 references that are in the map. First posted on arXiv 13 Aug 2023.

Builds on

48 of 51 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths29 May
  2. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models25 May
  3. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models25 May
  4. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  6. 2022Investigating Prompt Engineering in Diffusion Models21 Nov
  7. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Model15 Nov
  8. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
24 more
  1. 2016Layer Normalizationcited by 5,579
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  12. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  13. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  14. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  15. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  16. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  17. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  18. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  19. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  20. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  21. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  22. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  23. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
  24. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 48; 2024: 433; 2025: 597; 2026: 361.

1,439 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  2. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  3. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  4. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  5. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  6. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  7. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  8. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  9. 2024InstantID: Zero-shot Identity-Preserving Generation in Secondscited by 431
  10. 2024OmniGen: Unified Image Generationcited by 388
  11. 2025Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generationcited by 342
  12. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 334
38 more
  1. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  2. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 324
  3. 2024Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidancecited by 271
  4. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillationcited by 258
  5. 2023ImageDream: Image-Prompt Multi-view Diffusion for 3D Generationcited by 256
  6. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  7. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 240
  8. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
  9. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 225
  10. 2023Style Aligned Image Generation via Shared Attentioncited by 214
  11. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  12. 2024HQ-Edit: A High-Quality Dataset for Instruction-based Image Editingcited by 191
  13. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  14. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  15. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  16. 2024InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generationcited by 166
  17. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  18. 2024In-Context LoRA for Diffusion Transformerscited by 155
  19. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  20. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 153
  21. 2025Less-to-More Generalization: Unlocking More Controllability by In-Context Generationcited by 153
  22. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 152
  23. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  24. 2024OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-oncited by 150
  25. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
  26. 2024PuLID: Pure and Lightning ID Customization via Contrastive Alignmentcited by 143
  27. 2023SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generationcited by 142
  28. 2024MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidancecited by 142
  29. 2024Implicit Style-Content Separation using B-LoRAcited by 135
  30. 2024Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Imagecited by 132
  31. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  32. 2025Phantom: Subject-consistent video generation via cross-modal alignmentcited by 126
  33. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 124
  34. 2023Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Modelscited by 122
  35. 2024ID-Animator: Zero-Shot Identity-Preserving Human Video Generationcited by 122
  36. 2023Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Eraserscited by 118
  37. 2025EasyControl: Adding Efficient and Flexible Control for Diffusion Transformercited by 118
  38. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 115

50 most cited of 1,439 All 1,439 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023PRedItOR: Text Guided Image Editing with Diffusion Prior
  2. 2023The CLIP Model is Secretly an Image-to-Prompt Converter
  3. 2025Decoder-Only LLMs are Better Controllers for Diffusion Models
  4. 2025EDITOR: Effective and Interpretable Prompt Inversion for Text-to-Image Diffusion Models
  5. 2023CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Models
  6. 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models
  7. 2026i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models
  8. 2024Stylus: Automatic Adapter Selection for Diffusion Models