Data to 7 Oct 2026

Controlling Text-to-Image Diffusion by Orthogonal Finetuning

arXiv 2306.07280 ↗First posted 12 Jun 2023cs.CV cs.AI cs.GRcited by 207

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Large text-to-image diffusion models have impressive capabilities in generating photorealistic images from text prompts. How to effectively guide or control these powerful models to perform different downstream tasks becomes an important open problem. To tackle this challenge, we introduce a principled finetuning method -- Orthogonal Finetuning (OFT), for adapting text-to-image diffusion models to downstream tasks. Unlike existing methods, OFT can provably preserve hyperspherical energy which characterizes the pairwise neuron relationship on the unit hypersphere. We find that this property is crucial for preserving the semantic generation ability of text-to-image diffusion models. To improve finetuning stability, we further propose Constrained Orthogonal Finetuning (COFT) which imposes an additional radius constraint to the hypersphere. Specifically, we consider two important finetuning text-to-image tasks: subject-driven generation where the goal is to generate subject-specific images given a few images of a subject and a text prompt, and controllable generation where the goal is to enable the model to take in additional control signals. We empirically show that our OFT framework outperforms existing methods in generation quality and convergence speed.

Key facts. Cited by 207 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 65 of its 71 references that are in the map. First posted on arXiv 12 Jun 2023.

Builds on

65 of 71 references.

Closest ancestors · 10

Same direction, newest first

Other references · 55

2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2020Language Models are Few-Shot Learnerscited by 32,199
2020Denoising Diffusion Probabilistic Modelscited by 20,851
2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
2014Microsoft COCO: Common Objects in Contextcited by 17,498
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
2020Denoising Diffusion Implicit Modelscited by 9,655
2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339

38 more

2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
2021Zero-Shot Text-to-Image Generationcited by 4,276
2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
2016Generative Adversarial Text to Image Synthesiscited by 735
2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
2017Toward Multimodal Image-to-Image Translationcited by 295
2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
2019Controllable Text-to-Image Generationcited by 187
2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
2021Instance-Conditioned GANcited by 119
2018Learning towards Minimum Hyperspherical Energycited by 75
2017Deep Hyperspherical Learningcited by 55

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 76; 2025: 84; 2026: 36.

207 papers.

Sort:

By area:

2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
2024MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidancecited by 142
2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
2023TADA! Text to Animatable Digital Avatarscited by 109
2023TeCH: Text-guided Reconstruction of Lifelike Clothed Humanscited by 104
2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
2023DiffusionLight: Light Probes for Free by Painting a Chrome Ballcited by 63
2025SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learningcited by 60
2024SVFT: Parameter-Efficient Fine-Tuning with Singular Vectorscited by 57
2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steeringcited by 50
2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48

38 more

2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
2024ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preservingcited by 42
2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
2024Precise, Fast, and Low-cost Concept Erasure in Value Space: Orthogonal Complement Matterscited by 38
2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMscited by 36
2024Enhance Image Classification via Inter-Class Image Mixup with Diffusion Modelcited by 34
2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
2024Can Large Language Models Understand Symbolic Graphics Programs?cited by 32
2024Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNetscited by 30
2025Does FLUX Already Know How to Perform Physically Plausible Image Composition?cited by 30
2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
2024Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimizationcited by 25
2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 24
2024VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Modelscited by 24
2024ChatGarment: Garment Estimation, Generation and Editing via Large Language Modelscited by 22
2023NSF: Neural Surface Fields for Human Modeling from Monocular Depthcited by 21
2024Explorative Inbetweening of Time and Spacecited by 21
2024Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotationcited by 21
2024Spectral Adapter: Fine-Tuning in Spectral Spacecited by 21
2023CLiC: Concept Learning in Contextcited by 20
2025Personalized Image Generation with Deep Generative Models: A Decade Surveycited by 20
2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
2023A Data Perspective on Enhanced Identity Preservation for Diffusion Personalizationcited by 18
2024Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrivecited by 18
2024Face2Diffusion for Fast and Editable Face Personalizationcited by 17
2024Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Modelscited by 17
2024ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shiftscited by 16
2024ConceptExpress: Harnessing Diffusion Models for Single-image Unsupervised Concept Extractioncited by 16
2024Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptationcited by 15
2025T-LoRA: Single Image Diffusion Model Customization Without Overfittingcited by 15
2024CoRe: Context-Regularized Text Embedding Learning for Text-to-Image Personalizationcited by 14
2024Propulsion: Steering LLM with Tiny Fine-Tuningcited by 14
2024PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularizationcited by 14
2025Enhancing Reward Models for High-quality Image Generation: Beyond Text-Image Alignmentcited by 14
2025Value Gradient Guidance for Flow Matching Alignmentcited by 14
2024See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decompositioncited by 13
2025LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapterscited by 13
2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifoldcited by 13

50 most cited of 207 All 207 (JSON)

Similar, not cited

Close in wording, no citation either way