Polaris: Scaling Up Instruction-Guided Image Generation Towards Millions of Personalized Style Needs
Not on the map. Why?
Abstract (arXiv)
Users increasingly expect image generation models to quickly adapt to highly diverse and personalized requirements, such as producing images with distinctive styles or characteristics. Traditional approaches rely on fine-tuning, which is costly and difficult to scale. To cope with these limitations, the community has accumulated a growing library of fine-tuned modules and adapters, where each component targets specific generation needs and collectively serves as a foundation for handling new demands. This naturally raises a question: instead of repeatedly training new models, can we systematically exploit this expanding ecosystem to better fulfill user instructions? To this end, we present Polaris, an intelligent retrieval framework that automatically selects and integrates suitable models from the model library based on a user's instructions. The key insight is that harnessing such a massive and heterogeneous pool requires not only finding the most relevant modules among thousands of candidates, but also aligning them effectively for instruction-driven generation and editing. Polaris addresses this challenge by indexing over 6,500 checkpoints and 75,000 adapters, and retrieving the most relevant components given a user's input and instruction. In doing so, it delivers scalable, controllable, and well-aligned generation -- without any additional training.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 43 references that are in the map. First posted on arXiv 1 Jun 2026.
Builds on
39 of 43 references.
References · 39
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
- 2023Segment Anythingcited by 8,234
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2024GPT-4o System Cardcited by 4,588
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
27 more
- 2017Gradient Episodic Memory for Continual Learningcited by 1,898
- 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
- 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
- 2016HyperNetworkscited by 1,027
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
- 2023A Survey on Multimodal Large Language Modelscited by 909
- 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
- 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
- 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
- 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
- 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 429
- 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
- 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
- 2023Implicit Diffusion Models for Continuous Super-Resolutioncited by 218
- 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
- 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
- 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenizationcited by 87
- 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
- 2024DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representationscited by 68
- 2023ZONE: Zero-Shot Instruction-Guided Local Editingcited by 62
- 2021Model Zoo: A Growing "Brain" That Learns Continuallycited by 54
- 2023A Survey of Diffusion Based Image Generation Models: Issues and Their Solutionscited by 25
- 2025Personalized Image Generation with Deep Generative Models: A Decade Surveycited by 20
- 2024Stylus: Automatic Adapter Selection for Diffusion Modelscited by 18
- 2023A Survey of AI Text-to-Image and AI Text-to-Video Generatorscited by 17
- 2025A Model Zoo of Vision Transformerscited by 3
- 2025Varif.ai to Vary and Verify User-Driven Diversity in Scalable Image Generationcited by 1
Built on by
Not cited yet.