# Riemannian Optimization for Hadamard Products of Low-Rank Matrices

arXiv: https://arxiv.org/abs/2606.01216 · HTML: https://mlsanity.com/paper/2606.01216/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 28 references that are in the map.
- First posted on arXiv 31 May 2026.

## Builds on (most cited first, 10 of 10)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Manopt, a Matlab toolbox for optimization on manifolds](https://mlsanity.com/paper/1308.5200/) (2013)
- [Pymanopt: A Python Toolbox for Optimization on Manifolds using Automatic Differentiation](https://mlsanity.com/paper/1603.03236/) (2016)
- [Riemannian preconditioning](https://mlsanity.com/paper/1405.6055/) (2014)
- [R3MC: A Riemannian three-factor algorithm for low-rank matrix completion](https://mlsanity.com/paper/1306.2672/) (2013)
- [Efficient algorithms for the Hadamard decomposition](https://mlsanity.com/paper/2504.13633/) (2025)
- [ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models](https://mlsanity.com/paper/2505.14238/) (2025)
- [Manifold-based Algorithms for the Hadamard Decomposition](https://mlsanity.com/paper/2605.28980/) (2026)
- [BoHA: Blockwise Hadamard Product Adaptation for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2509.21637/) (2025)

## Abstract (arXiv)

The elementwise Hadamard product of two low-rank matrices provides a parameter-efficient model for data with multiplicative structure, but its modeling is challenging due to the presence of additional symmetries under coupled row/column scalings between the two factors. In order to leverage the geometry of the space, we formulate the learning of such matrices as optimization on a Riemannian quotient manifold. We propose a novel block-diagonal Riemannian metric derived from the pullback of the Frobenius inner product. The metric is shown to be invariant under these symmetries. We develop a Riemannian gradient descent algorithm that uses a tuning-free Gauss--Newton step size and scales linearly in the number of observed entries per iteration. The versatile framework of Riemannian quotient optimization enables both first-order and second-order Riemannian methods, the latter through a closed-form connection and the Riemannian Hessian. Experiments on real and synthetic datasets illustrate the efficacy of our proposed Riemannian approach.