# X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model arXiv: https://arxiv.org/abs/2312.02238 ยท HTML: https://mlsanity.com/paper/2312.02238/ ## Key facts - Cited by 19 papers in the map (data through 14 Oct 2026). - Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. - Builds on 27 of its 37 references that are in the map. - First posted on arXiv 4 Dec 2023. ## Placed in - Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/) ## Builds on (most cited first, 27 of 27) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021) - [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021) - [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015) - [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022) - [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023) - [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023) - [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021) - [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022) - [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022) - [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021) - [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023) - [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023) - [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023) - [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019) - [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021) - [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023) - [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023) - [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023) - [Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2309.15818/) (2023) - [Inserting Anybody in Diffusion Models via Celeb Basis](https://mlsanity.com/paper/2306.00926/) (2023) - [Revisit Parameter-Efficient Transfer Learning: A Two-Stage Paradigm](https://mlsanity.com/paper/2303.07910/) (2023) - [TaCA: Upgrading Your Visual Foundation Model with Task-agnostic Compatible Adapter](https://mlsanity.com/paper/2306.12642/) (2023) ## Built on by (most cited first, 19 of 19) - [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024) - [Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model](https://mlsanity.com/paper/2404.09967/) (2024) - [ControlVAR: Exploring Controllable Visual Autoregressive Modeling](https://mlsanity.com/paper/2406.09750/) (2024) - [Parameter-Efficient Fine-Tuning for Foundation Models](https://mlsanity.com/paper/2501.13787/) (2025) - [CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Design](https://mlsanity.com/paper/2505.19114/) (2025) - [Conditional Image Synthesis with Diffusion Models: A Survey](https://mlsanity.com/paper/2409.19365/) (2024) - [CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation](https://mlsanity.com/paper/2410.09400/) (2024) - [SCALAR: Scale-wise Controllable Visual Autoregressive Learning](https://mlsanity.com/paper/2507.19946/) (2025) - [MUSCLE: A Model Update Strategy for Compatible LLM Evolution](https://mlsanity.com/paper/2407.09435/) (2024) - [LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation](https://mlsanity.com/paper/2501.16559/) (2025) - [TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision](https://mlsanity.com/paper/2503.07416/) (2025) - [Zero-Shot Adaptation of Parameter-Efficient Fine-Tuning in Diffusion Models](https://mlsanity.com/paper/2506.04244/) (2025) - [Exploring Data-Free LoRA Transferability for Video Diffusion Models](https://mlsanity.com/paper/2605.01929/) (2026) - [Not All Parameters Matter: Masking Diffusion Models for Enhancing Generation Ability](https://mlsanity.com/paper/2505.03097/) (2025) - [Glance: Accelerating Diffusion Models with 1 Sample](https://mlsanity.com/paper/2512.02899/) (2025) - [Enhancing Hands in 3D Whole-Body Pose Estimation with Conditional Hands Modulator](https://mlsanity.com/paper/2603.14726/) (2026) - [Heeding the Inner Voice: Aligning ControlNet Training via Intermediate Features Feedback](https://mlsanity.com/paper/2507.02321/) (2025) - [No Re-Train, More Gain: Upgrading Backbones with Diffusion model for Pixel-Wise and Weakly-Supervised Few-Shot Segmentation](https://mlsanity.com/paper/2407.16182/) (2024) - [DART: Distillation-Aware Reparameterization for Training-Free LoRA Reuse in Few-Step Video Diffusion Models](https://mlsanity.com/paper/2609.20051/) (2026) ## Abstract (arXiv) We introduce X-Adapter, a universal upgrader to enable the pretrained plug-and-play modules (e.g., ControlNet, LoRA) to work directly with the upgraded text-to-image diffusion model (e.g., SDXL) without further retraining. We achieve this goal by training an additional network to control the frozen upgraded model with the new text-image data pairs. In detail, X-Adapter keeps a frozen copy of the old model to preserve the connectors of different plugins. Additionally, X-Adapter adds trainable mapping layers that bridge the decoders from models of different versions for feature remapping. The remapped features will be used as guidance for the upgraded model. To enhance the guidance ability of X-Adapter, we employ a null-text training strategy for the upgraded model. After training, we also introduce a two-stage denoising strategy to align the initial latents of X-Adapter and the upgraded model. Thanks to our strategies, X-Adapter demonstrates universal compatibility with various plugins and also enables plugins of different versions to work together, thereby expanding the functionalities of diffusion community. To verify the effectiveness of the proposed method, we conduct extensive experiments and the results show that X-Adapter may facilitate wider application in the upgraded foundational diffusion model.