mlsanityData to 14 Oct 2026

Ortho-Hydra: Orthogonalized Experts for DiT LoRA

arXiv 2605.03252 ↗First posted 5 May 2026cs.LG cs.AI cs.CVcited by 0

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

LoRA fine-tuning of diffusion transformers (DiT) on multi-style data suffers from style bleed: a single low-rank residual cannot represent several distinct artist fingerprints, and the optimizer converges to their average. Mixture-of-experts LoRA in the HydraLoRA style replaces the up-projection with E heads under a router, but when every expert is zero-initialized the router receives identical gradient from each head and remains at the uniform prior. The experts then evolve permutation-symmetrically, and the network trains as a single rank-r LoRA at E× the cost. We present Ortho-Hydra, a re-parameterisation that combines an OFT-style Cayley-orthogonal shared basis with per-expert disjoint output subspaces carved from the top-(Er) left singular vectors of the pretrained weight. Disjointness makes the router's per-expert score non-degenerate at step~0, so specialization receives gradient signal before any expert has trained. We test the predicted deadlock on a DiT pipeline by comparing two HydraLoRA baselines, a zero-initialized shared-basis variant and the original σ=0.1 Gaussian-jitter mitigation, against Ortho-Hydra under a matched optimiser, dataset, and step budget. Neither baseline leaves the uniform prior within the first 1k steps; Ortho-Hydra begins de-uniformising within the first few hundred. End-task generation quality on multi-style data is out of scope; we report the construction, the cold-start mechanism, and the routing dynamics it changes. Code: https://github.com/sorryhyun/animalora.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 7 of its 9 references that are in the map. First posted on arXiv 5 May 2026.

Builds on

7 of 9 references.

References · 7

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 208
  4. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 134
  5. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 114
  6. 2025Advancing Expert Specialization for Better MoEcited by 39
  7. 2025Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MoECa: Aligning Feature Reuse with Expert Decomposition in Diffusion Transformers
  2. 2025Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts
  3. 2024Scaling Diffusion Transformers to 16 Billion Parameters
  4. 2025TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
  5. 2024Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoising
  6. 2026Sparse Mixture-of-Experts Routing in Visual Diffusion Transformers:Diagnosis, Boundary Calibration and Evolutionary Roadmap from Routing Collapse to Selective Deadlock
  7. 2026Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Experts
  8. 2025OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuning