# The EDelta-MHC-Geo Transformer: Adaptive Geodesic Operations with Guaranteed Orthogonality

arXiv: https://arxiv.org/abs/2605.06729 · HTML: https://mlsanity.com/paper/2605.06729/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 15 references that are in the map.
- First posted on arXiv 7 May 2026.

## Builds on (most cited first, 8 of 8)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group](https://mlsanity.com/paper/1901.08428/) (2019)
- [Hyper-Connections](https://mlsanity.com/paper/2409.19606/) (2024)
- [On orthogonality and learning recurrent networks with long term dependencies](https://mlsanity.com/paper/1702.00071/) (2017)
- [Orthogonal Recurrent Neural Networks with Scaled Cayley Transform](https://mlsanity.com/paper/1707.09520/) (2017)
- [JPmHC Dynamical Isometry via Orthogonal Hyper-Connections](https://mlsanity.com/paper/2602.18308/) (2026)

## Abstract (arXiv)

We present the EDelta-MHC-Geo Transformer, a novel architecture that unifies Manifold-Constrained Hyper-Connections (mHC), Deep Delta Learning (DDL), and the Cayley transform to obtain input-adaptive, unconditionally orthogonal residual connections. Unlike DDL, whose Householder operator is orthogonal only at β ∈ 0,2, our Data-Dependent Cayley rotation Q(x)=(I+(β/2)A(x))-1(I-(β/2)A(x)) preserves orthogonality for all β and all inputs. To handle negation, an eigenvalue -1 case that Cayley provably excludes, we introduce the EDelta-MHC-Geo Hybrid, which combines Cayley rotation with Householder reflection via a learned operator-selection gate X'=γ(X)Q(X)X+(1-γ(X))H2(X)X. A midpoint-collapse regularizer, 4γ(1-γ), encourages boundary gate decisions, where each selected component is orthogonal. In matched-parameter comparisons, with approximately 1.79M parameters per model and mean +/- standard deviation over 3 seeds, against four baselines including the concurrent JPmHC, EDelta-MHC-Geo achieves the best long-horizon stability, 1.9x over JPmHC and 3.8x over GPT; the best near-π rotation loss, 4.5x over JPmHC on single-plane; strong norm preservation, with 0.001 mean deviation; and 0.96 negation cosine alignment in a diagnostic reflection probe, all with 33% fewer layers. While JPmHC's wider representation excels on pure rotation, its finite Cayley residual mixer excludes an exact λ=-1 operator and has no reflection branch, motivating our hybrid approach for accessing both connected components of O(n).