mlsanityData to 14 Oct 2026

LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Models

arXiv 2609.25884 ↗First posted 22 Sep 2026cs.CR cs.CVcited by 0

On the map: Diffusion models → Robustness, privacy and adversarial aspects

Abstract (arXiv)

Users commonly combine multiple Low-Rank Adaptation (LoRA) adapters to personalize images with different subjects, styles, and visual attributes. Yet inspecting adapters individually does not establish the safety of their composition. We identify and characterize a pair-conditioned attack in text-to-image diffusion: individually useful and benign-appearing adapters redirect image generation when co-loaded with a specifically matched partner, whose identity serves as the trigger. We introduce LoRango to realize this attack through complementary Signature and Payload adapters. The Signature writes a pair-specific code into intermediate carrier representations, while the Payload uses code-selective responses and opposing signal/reference branches. These branches approximately cancel for standalone adapters and mismatched pairs; matched code-reader alignment breaks cancellation within native GEGLU blocks and releases the programmed action. Both adapters are exported as ordinary static LoRA files compatible with standard loaders, requiring no prompt trigger or base-pipeline modification. LoRango achieves matched-pair attack success rates of 97.9% on SD v1.5 and 98.7% on SDXL, compared with 2.8--4.6% when implanted adapters are loaded individually. Further experiments evaluate pair selectivity, standalone fidelity, robustness to deployment variations, and applicability across denoiser architectures. These findings show that individual-adapter inspection is insufficient to assess the security of multi-LoRA personalization and motivate auditing adapter compositions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 34 of its 39 references that are in the map. First posted on arXiv 22 Sep 2026.

Builds on

34 of 39 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026When LoRA Betrays: Backdooring Text-to-Image Models by Masquerading as Benign Adapters25 Feb
  2. 2025BackdoorDM: A Comprehensive Benchmark for Backdoor Learning on Diffusion Model17 Feb
  3. 2023VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Models12 Jun
  4. 2023Personalization as a Shortcut for Few-Shot Backdoor Attack against Text-to-Image Diffusion Models18 May
  5. 2023Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning7 May
  6. 2023TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targets10 Mar
  7. 2022How to Backdoor Diffusion Models?11 Dec

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  10. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  11. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  12. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,106
15 more
  1. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
  2. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  3. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondencecited by 214
  4. 2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 143
  5. 2023Diffusion Model as Representation Learnercited by 95
  6. 2023Orthogonal Adaptation for Modular Customization of Diffusion Modelscited by 70
  7. 2022Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesiscited by 59
  8. 2023BAGM: A Backdoor Attack for Manipulating Text-to-Image Generative Modelscited by 37
  9. 2023Beyond Surface Statistics: Scene Representations in a Latent Diffusion Modelcited by 30
  10. 2024LoRACLR: Contrastive Adaptation for Customization of Diffusion Modelscited by 20
  11. 2025T-LoRA: Single Image Diffusion Model Customization Without Overfittingcited by 15
  12. 2024Contrastive Test-Time Composition of Multiple LoRA Models for Image Generationcited by 10
  13. 2025AutoLoRA: Automatic LoRA Retrieval and Fine-Grained Gated Fusion for Text-to-Image Generationcited by 5
  14. 2025TC-LoRA: Temporally Modulated Conditional LoRA for Adaptive Diffusion Controlcited by 5
  15. 2026Colluding LoRA: A Compositional Vulnerability in LLM Safety Alignmentcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026LoRA-Key: User-Centric LoRA Watermarking for Text-to-Image Diffusion Models
  2. 2025Semantic Mismatch and Perceptual Degradation: A New Perspective on Image Editing Immunity
  3. 2025LoRAtorio: An intrinsic approach to LoRA Skill Composition
  4. 2026Diffu-LoRA: A Novel Low-Rank Adaptation for Personalized Diffusion Models
  5. 2025LoRAShield: Data-Free Editing Alignment for Secure Personalized LoRA Sharing
  6. 2025PersGuard: Preventing Malicious Personalization in Text-to-Image Diffusion Models via Model Backdoors
  7. 2024Dual-Model Defense: Safeguarding Diffusion Models from Membership Inference Attacks through Disjoint Data Splitting
  8. 2026Discovering Undisclosed Concepts in LoRA Adapters for Diffusion-Based AIGC Models