# Riemannian Structure and Optimization for a Class of Low-Parametric Orthogonal Matrices

arXiv: https://arxiv.org/abs/2609.27982 · HTML: https://mlsanity.com/paper/2609.27982/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 27 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Builds on (most cited first, 11 of 11)

- [Controlling Text-to-Image Diffusion by Orthogonal Finetuning](https://mlsanity.com/paper/2306.07280/) (2023)
- [Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization](https://mlsanity.com/paper/2311.06243/) (2023)
- [Monarch: Expressive Structured Matrices for Efficient and Accurate Training](https://mlsanity.com/paper/2204.00595/) (2022)
- [Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks](https://mlsanity.com/paper/1911.00937/) (2019)
- [Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations](https://mlsanity.com/paper/1903.05895/) (2019)
- [Skew Orthogonal Convolutions](https://mlsanity.com/paper/2105.11417/) (2021)
- [Structured Transforms for Small-Footprint Deep Learning](https://mlsanity.com/paper/1510.01722/) (2015)
- [Group and Shuffle: Efficient Structured Orthogonal Parametrization](https://mlsanity.com/paper/2406.10019/) (2024)
- [Automatic differentiation for Riemannian optimization on low-rank matrix and tensor-train manifolds](https://mlsanity.com/paper/2103.14974/) (2021)
- [The tangent cone to the real determinantal variety: various expressions and a proof](https://mlsanity.com/paper/2504.11382/) (2025)
- [OrthoFuse: Training-free Riemannian Fusion of Orthogonal Style-Concept Adapters for Diffusion Models](https://mlsanity.com/paper/2604.05183/) (2026)

## Abstract (arXiv)

In this paper, we are concerned with matrices formed by block-diagonal factors interleaved with fixed permutations -- a flexible family of structured matrices. This class has recently drawn interest in deep learning architectures for its balanced expressivity-efficiency trade-off, yet efficient computational strategies for working with it remain to be found. We approach this problem through Riemannian geometry and examine under what conditions this class admits a smooth manifold structure. For the practically important case of orthogonal two-factor matrices, we derive the essential Riemannian tools and propose efficient algorithms for their implementation. The algorithms leverage automatic differentiation, support parameter sharing within each factor, and avoid explicit dense matrix construction. We test them within the Riemannian optimization framework on the best matrix approximation problem and for parameter-efficient fine-tuning of large language models. Beyond the two-factor setting, we study the geometric and matrix-theoretic properties of factorizations with a larger number of block-diagonal factors.