Data to 7 Oct 2026

Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation

arXiv 2309.14859 ↗First posted 26 Sep 2023cs.CV cs.AI cs.GRcited by 108

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-to-image generative models have garnered immense attention for their ability to produce high-fidelity images from text prompts. Among these, Stable Diffusion distinguishes itself as a leading open-source model in this fast-growing field. However, the intricacies of fine-tuning these models pose multiple challenges from new methodology integration to systematic evaluation. Addressing these issues, this paper introduces LyCORIS (Lora beYond Conventional methods, Other Rank adaptation Implementations for Stable diffusion) [https://github.com/KohakuBlueleaf/LyCORIS], an open-source library that offers a wide selection of fine-tuning methodologies for Stable Diffusion. Furthermore, we present a thorough framework for the systematic assessment of varied fine-tuning techniques. This framework employs a diverse suite of metrics and delves into multiple facets of fine-tuning, including hyperparameter adjustments and the evaluation with different prompt types across various concept categories. Through this comprehensive approach, our work provides essential insights into the nuanced effects of fine-tuning parameters, bridging the gap between state-of-the-art research and practical application.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 55 of its 103 references that are in the map. First posted on arXiv 26 Sep 2023.

Builds on

55 of 103 references.

Closest ancestors · 17

Same direction, newest first

5 more

Other references · 38

2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2020Denoising Diffusion Probabilistic Modelscited by 20,851
2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2020Denoising Diffusion Implicit Modelscited by 9,655
2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
2017A Unified Approach to Interpreting Model Predictionscited by 6,488
2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
2017Decoupled Weight Decay Regularizationcited by 1,722

26 more

2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 414
20218-bit Optimizers via Block-wise Quantizationcited by 395
2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 299
2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
2023HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Modelscited by 224
2017GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibriumcited by 208
2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 146
2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 140
2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 130
2023A Neural Space-Time Representation for Text-to-Image Personalizationcited by 123
2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 84
2020Few-Shot Adaptation of Generative Adversarial Networkscited by 64
2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
2022Human Evaluation of Text-to-Image Models on a Multi-Task Benchmarkcited by 41
2021DAF:re: A Challenging, Crowd-Sourced, Large-Scale, Long-Tailed Dataset For Anime Character Recognitioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 45; 2025: 40; 2026: 22.

108 papers.

Sort:

By area:

2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
2024Controlling Language and Diffusion Models by Transporting Activationscited by 59
2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
2024Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancycited by 38
2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMscited by 36
2024Customizing Language Models with Instance-wise LoRA for Sequential Recommendationcited by 32

38 more

2024WildFake: A Large-scale Challenging Dataset for AI-Generated Images Detectioncited by 31
2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
2023Diffusion Illusions: Hiding Images in Plain Sightcited by 26
2024Edify Image: High-Quality Image Generation with Pixel Space Laplacian Diffusion Modelscited by 22
2024ResLoRA: Identity Residual Mapping in Low-Rank Adaptioncited by 18
2024See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decompositioncited by 13
2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13
2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Modelscited by 13
2024DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Modelcited by 12
2025Ultra-Resolution Adaptation with Easecited by 12
2024DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion Modelscited by 11
2024Large Convolutional Model Tuning via Filter Subspacecited by 11
2024DiffLoRA: Generating Personalized Low-Rank Adaptation Weights with Diffusioncited by 11
2024MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mambacited by 11
2024SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modulescited by 10
2024PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reductioncited by 10
2024TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competitioncited by 10
2025HyperLoRA: Parameter-Efficient Adaptive Generation for Portrait Synthesiscited by 10
2024Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgettingcited by 9
2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 9
2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Modelscited by 9
2024TPP-LLM: Modeling Temporal Point Processes by Efficiently Fine-Tuning Large Language Modelscited by 8
2024COAP: Memory-Efficient Training with Correlation-Aware Gradient Projectioncited by 8
2025Adapting Vision Foundation Models for Real-time Ultrasound Image Segmentationcited by 8
2024Block-wise LoRA: Revisiting Fine-grained LoRA for Effective Personalization and Stylization in Text-to-Image Generationcited by 7
2025Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Datacited by 7
2025Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Visioncited by 7
2026ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Taskscited by 6
2024CAT: Contrastive Adapter Training for Personalized Image Generationcited by 5
2024Illustrious: an Open Advanced Illustration Modelcited by 5
2024FedPT: Federated Proxy-Tuning of Large Language Models on Resource-Constrained Edge Devicescited by 5
2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5
2024CTRLorALTer: Conditional LoRAdapter for Efficient 0-Shot Control & Altering of T2I Modelscited by 4
2024Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuningcited by 4
2024Task-Specific Directions: Definition, Exploration, and Utilization in Parameter Efficient Fine-Tuningcited by 4
2024Efficient Source-Free Time-Series Adaptation via Parameter Subspace Disentanglementcited by 4
2025QuIC: Quantum-Inspired Compound Adapters for Parameter Efficient Fine-Tuningcited by 4
2025DenseLoRA: Dense Low-Rank Adaptation of Large Language Modelscited by 4

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way