mlsanityData to 14 Oct 2026

Polaris: Scaling Up Instruction-Guided Image Generation Towards Millions of Personalized Style Needs

arXiv 2606.01858 ↗First posted 1 Jun 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Users increasingly expect image generation models to quickly adapt to highly diverse and personalized requirements, such as producing images with distinctive styles or characteristics. Traditional approaches rely on fine-tuning, which is costly and difficult to scale. To cope with these limitations, the community has accumulated a growing library of fine-tuned modules and adapters, where each component targets specific generation needs and collectively serves as a foundation for handling new demands. This naturally raises a question: instead of repeatedly training new models, can we systematically exploit this expanding ecosystem to better fulfill user instructions? To this end, we present Polaris, an intelligent retrieval framework that automatically selects and integrates suitable models from the model library based on a user's instructions. The key insight is that harnessing such a massive and heterogeneous pool requires not only finding the most relevant modules among thousands of candidates, but also aligning them effectively for instruction-driven generation and editing. Polaris addresses this challenge by indexing over 6,500 checkpoints and 75,000 adapters, and retrieving the most relevant components given a user's input and instruction. In doing so, it delivers scalable, controllable, and well-aligned generation -- without any additional training.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 43 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

39 of 43 references.

References · 39

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2023Segment Anythingcited by 8,234
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
27 more
  1. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  2. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  3. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  4. 2016HyperNetworkscited by 1,027
  5. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  6. 2023A Survey on Multimodal Large Language Modelscited by 909
  7. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  8. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  9. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  10. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  11. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 429
  12. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  13. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  14. 2023Implicit Diffusion Models for Continuous Super-Resolutioncited by 218
  15. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
  16. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  17. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenizationcited by 87
  18. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  19. 2024DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representationscited by 68
  20. 2023ZONE: Zero-Shot Instruction-Guided Local Editingcited by 62
  21. 2021Model Zoo: A Growing "Brain" That Learns Continuallycited by 54
  22. 2023A Survey of Diffusion Based Image Generation Models: Issues and Their Solutionscited by 25
  23. 2025Personalized Image Generation with Deep Generative Models: A Decade Surveycited by 20
  24. 2024Stylus: Automatic Adapter Selection for Diffusion Modelscited by 18
  25. 2023A Survey of AI Text-to-Image and AI Text-to-Video Generatorscited by 17
  26. 2025A Model Zoo of Vision Transformerscited by 3
  27. 2025Varif.ai to Vary and Verify User-Driven Diversity in Scalable Image Generationcited by 1

Built on by

Not cited yet.