mlsanityData to 14 Oct 2026

Stylus: Automatic Adapter Selection for Diffusion Models

arXiv 2404.18928 ↗First posted 29 Apr 2024cs.CV cs.AI cs.CLcited by 18

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Beyond scaling base models with more data or parameters, fine-tuned adapters provide an alternative way to generate high fidelity, custom images at reduced costs. As such, adapters have been widely adopted by open-source communities, accumulating a database of over 100K adapters-most of which are highly customized with insufficient descriptions. This paper explores the problem of matching the prompt to a set of relevant adapters, built on recent work that highlight the performance gains of composing adapters. We introduce Stylus, which efficiently selects and automatically composes task-specific adapters based on a prompt's keywords. Stylus outlines a three-stage approach that first summarizes adapters with improved descriptions and embeddings, retrieves relevant adapters, and then further assembles adapters based on prompts' keywords by checking how well they fit the prompt. To evaluate Stylus, we developed StylusDocs, a curated dataset featuring 75K adapters with pre-computed adapter embeddings. In our evaluation on popular Stable Diffusion checkpoints, Stylus achieves greater CLIP-FID Pareto efficiency and is twice as preferred, with humans and multimodal models as evaluators, over the base model. See stylus-diffusion.github.io for more.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 34 of its 48 references that are in the map. First posted on arXiv 29 Apr 2024.

Builds on

34 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models20 Nov
  2. 2023Effective Data Augmentation With Diffusion Models7 Feb

Other references · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
20 more
  1. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  3. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  4. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  6. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  7. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 434
  8. 2018Assessing Generative Models via Precision and Recallcited by 406
  9. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  10. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  11. 2023Fine-tuning Language Models for Factualitycited by 221
  12. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  13. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 73
  14. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  15. 2023Describing Differences in Image Sets with Natural Languagecited by 53
  16. 2024LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Taskscited by 39
  17. 2023Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Modelscited by 36
  18. 2024Routoo: Learning to Route to Large Language Models Effectivelycited by 34
  19. 2021Efficient Test Time Adapter Ensembling for Low-resource Language Varietiescited by 26
  20. 2023Vector Search with OpenAI Embeddings: Lucene Is All You Needcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 9; 2026: 5.

18 papers.

Sort:

By area:

  1. 2025We Should Chart an Atlas of All the World's Modelscited by 25
  2. 2025Deepfakes on Demand: the rise of accessible non-consensual deepfake image generatorscited by 15
  3. 2024Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learningcited by 14
  4. 2025LinEAS: End-to-end Learning of Activation Steering with a Distributional Losscited by 14
  5. 2024PAK-UCB Contextual Bandit: An Online Learning Approach to Prompt-Aware Selection of Generative Models and LLMscited by 12
  6. 2025LACONIC: A 3D Layout Adapter for Controllable Image Creationcited by 6
  7. 2026PersonalQ: Select, Quantize, and Serve Personalized Diffusion Models for Efficient Inferencecited by 6
  8. 2025Preventing Shortcuts in Adapter Training via Providing the Shortcutscited by 4
  9. 2025LoRAverse: A Submodular Framework to Retrieve Diverse Adapters for Diffusion Modelscited by 3
  10. 2024Beyond Adapter Retrieval: Latent Geometry-Preserving Composition via Sparse Task Projectioncited by 2
  11. 2026tLoRA: Efficient Multi-LoRA Training with Elastic Shared Super-Modelscited by 2
  12. 2026Discovering Hidden Gems in Model Repositoriescited by 1
6 more
  1. 2024MANTA -- Model Adapter Native generations that's Affordablecited by 0
  2. 2025CGI: Identifying Conditional Generative Models with Example Imagescited by 0
  3. 2025Delta Activations: A Representation for Finetuned Large Language Modelscited by 0
  4. 2025CARLoS: Retrieval via Concise Assessment Representation of LoRAs at Scalecited by 0
  5. 2026GimmBO: Interactive Generative Image Model Merging via Bayesian Optimizationcited by 0
  6. 2026Polaris: Scaling Up Instruction-Guided Image Generation Towards Millions of Personalized Style Needscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models
  2. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models
  3. 2023Continual Diffusion with STAMINA: STack-And-Mask INcremental Adapters
  4. 2025Steer Away From Mode Collisions: Improving Composition In Diffusion Models
  5. 2023Prompt-Propose-Verify: A Reliable Hand-Object-Interaction Data Generation Framework using Foundational Models
  6. 2026i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models
  7. 2026Evolutionary Token-Level Prompt Optimization for Diffusion Models
  8. 2025Inference-Time Optimization of Prompt Embeddings in Diffusion Models: A Comparison of sep-CMA-ES and Adam