mlsanityData to 14 Oct 2026

Deep Network Trainability via Persistent Subspace Orthogonality

arXiv 2508.02882 ↗First posted 4 Aug 2025cs.LG cs.NA math.NAcited by 0

Not on the map. Why?

Abstract (arXiv)

Training neural networks via backpropagation is often hindered by vanishing or exploding gradients. In this work, we design architectures that mitigate these issues by analyzing and controlling the network Jacobian. We first provide a unified characterization for a class of networks with orthogonal Jacobian including known architectures and yielding new trainable designs. We then introduce the relaxed notion of persistent subspace orthogonality. This applies to a broader class of networks whose Jacobians are isometries only on a non-trivial subspace. We propose practical mechanisms to enforce this condition and empirically show that it is necessary to sufficiently preserve the gradient norms during backpropagation, enabling the training of very deep networks. We support our theory with extensive experiments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 53 references that are in the map. First posted on arXiv 4 Aug 2025.

Builds on

34 of 53 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2016Layer Normalizationcited by 5,669
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2017Improved Training of Wasserstein GANscited by 2,235
  6. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  7. 2018Group Normalizationcited by 1,524
  8. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  9. 2017Self-Normalizing Neural Networkscited by 598
  10. 2016Error bounds for approximations with deep ReLU networkscited by 574
  11. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 519
  12. 2017Stable Architectures for Deep Neural Networkscited by 341
22 more
  1. 2015Highway Networkscited by 260
  2. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 231
  3. 2018Sorting out Lipschitz function approximationcited by 213
  4. 2019Plug-and-Play Methods Provably Converge with Properly Trained Denoiserscited by 198
  5. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 195
  6. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  7. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  8. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  9. 2019Orthogonal Convolutional Neural Networkscited by 123
  10. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  11. 2018Adversarial Regularizers in Inverse Problemscited by 103
  12. 2021Skew Orthogonal Convolutionscited by 67
  13. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 60
  14. 2020Convolutional Proximal Neural Networks and Plug-and-Play Algorithmscited by 36
  15. 2015Deep Learning with S-shaped Rectified Linear Activation Unitscited by 26
  16. 2021Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Designcited by 18
  17. 20231-Lipschitz Layers Compared: Memory, Speed, and Certifiable Robustnesscited by 16
  18. 2023Designing Stable Neural Networks using Convex Analysis and ODEscited by 10
  19. 2021Learning specialized activation functions with the Piecewise Linear Unitcited by 7
  20. 2025Approximation theory for 1-Lipschitz ResNetscited by 7
  21. 2024Symplectic Methods in Deep Learningcited by 3
  22. 2025On the expressivity of deep Heaviside networkscited by 3

Built on by

Not cited yet.