# Structured Transformations for Stable and Interpretable Neural Computation

arXiv: https://arxiv.org/abs/2508.00127 · HTML: https://mlsanity.com/paper/2508.00127/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 26 references that are in the map.
- First posted on arXiv 31 Jul 2025.

## Builds on (most cited first, 23 of 23)

- [Small-scale proxies for large-scale Transformer training instabilities](https://mlsanity.com/paper/2309.14322/) (2023)
- [Understanding plasticity in neural networks](https://mlsanity.com/paper/2303.01486/) (2023)
- [Stabilizing Transformer Training by Preventing Attention Entropy Collapse](https://mlsanity.com/paper/2303.06296/) (2023)
- [Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse](https://mlsanity.com/paper/2206.03126/) (2022)
- [How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding](https://mlsanity.com/paper/2303.04245/) (2023)
- [Disentangling the Causes of Plasticity Loss in Neural Networks](https://mlsanity.com/paper/2402.18762/) (2024)
- [Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs](https://mlsanity.com/paper/2206.00939/) (2022)
- [PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks](https://mlsanity.com/paper/2402.00326/) (2024)
- [Anti-Symmetric DGN: a stable architecture for Deep Graph Networks](https://mlsanity.com/paper/2210.09789/) (2022)
- [Implicit Regularization Towards Rank Minimization in ReLU Networks](https://mlsanity.com/paper/2201.12760/) (2022)
- [Learning threshold neurons via the "edge of stability"](https://mlsanity.com/paper/2212.07469/) (2022)
- [Direct Parameterization of Lipschitz-Bounded Deep Networks](https://mlsanity.com/paper/2301.11526/) (2023)
- [Break It Down: Evidence for Structural Compositionality in Neural Networks](https://mlsanity.com/paper/2301.10884/) (2023)
- [The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descent](https://mlsanity.com/paper/2305.17490/) (2023)
- [The Neural Race Reduction: Dynamics of Abstraction in Gated Networks](https://mlsanity.com/paper/2207.10430/) (2022)
- [A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases](https://mlsanity.com/paper/2209.11208/) (2022)
- [Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks](https://mlsanity.com/paper/2201.11729/) (2022)
- [On the Versatile Uses of Partial Distance Correlation in Deep Learning](https://mlsanity.com/paper/2207.09684/) (2022)
- [Can Transformers Learn to Solve Problems Recursively?](https://mlsanity.com/paper/2305.14699/) (2023)
- [The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks](https://mlsanity.com/paper/2306.17499/) (2023)
- [Singular Value Perturbation and Deep Network Optimization](https://mlsanity.com/paper/2203.03099/) (2022)
- [Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks](https://mlsanity.com/paper/2305.15912/) (2023)
- [Neural Network Optimal Feedback Control with Guaranteed Local Stability](https://mlsanity.com/paper/2205.00394/) (2022)

## Built on by (most cited first, 2 of 2)

- [Cross-Model Semantics in Representation Learning](https://mlsanity.com/paper/2508.03649/) (2025)
- [Understanding Learning Dynamics Through Structured Representations](https://mlsanity.com/paper/2508.02126/) (2025)

## Abstract (arXiv)

Despite their impressive performance, contemporary neural networks often lack structural safeguards that promote stable learning and interpretable behavior. In this work, we introduce a reformulation of layer-level transformations that departs from the standard unconstrained affine paradigm. Each transformation is decomposed into a structured linear operator and a residual corrective component, enabling more disciplined signal propagation and improved training dynamics. Our formulation encourages internal consistency and supports stable information flow across depth, while remaining fully compatible with standard learning objectives and backpropagation. Through a series of synthetic and real-world experiments, we demonstrate that models constructed with these structured transformations exhibit improved gradient conditioning, reduced sensitivity to perturbations, and layer-wise robustness. We further show that these benefits persist across architectural scales and training regimes. This study serves as a foundation for a more principled class of neural architectures that prioritize stability and transparency-offering new tools for reasoning about learning behavior without sacrificing expressive power.