# Optimization on Submanifolds of Convolution Kernels in CNNs

arXiv: https://arxiv.org/abs/1610.07008 · HTML: https://mlsanity.com/paper/1610.07008/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2016.

## Built on by (most cited first, 15 of 15)

- [Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts](https://mlsanity.com/paper/2311.11385/) (2023)
- [Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows](https://mlsanity.com/paper/2106.05275/) (2021)
- [A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold](https://mlsanity.com/paper/2302.08210/) (2023)
- [Mano: Restriking Manifold Optimization for LLM Training](https://mlsanity.com/paper/2601.23000/) (2026)
- [Dynamical systems' based neural networks](https://mlsanity.com/paper/2210.02373/) (2022)
- [boldsymbolλ-Orthogonality Regularization for Compatible Representation Learning](https://mlsanity.com/paper/2509.16664/) (2025)
- [Coordinate descent on the orthogonal group for recurrent neural network training](https://mlsanity.com/paper/2108.00051/) (2021)
- [Entangled Residual Mappings](https://mlsanity.com/paper/2206.01261/) (2022)
- [MCNC: Manifold-Constrained Reparameterization for Neural Compression](https://mlsanity.com/paper/2406.19301/) (2024)
- [Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networks](https://mlsanity.com/paper/2103.06583/) (2021)
- [Ortho-Shot: Low Displacement Rank Regularization with Data Augmentation for Few-Shot Learning](https://mlsanity.com/paper/2110.09374/) (2021)
- [Towards Better Orthogonality Regularization with Disentangled Norm in Training Deep CNNs](https://mlsanity.com/paper/2306.09939/) (2023)
- [An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scale](https://mlsanity.com/paper/2602.14656/) (2026)
- [Group Orthogonalization Regularization For Vision Models Adaptation and Robustness](https://mlsanity.com/paper/2306.10001/) (2023)
- [Parallelized Computation and Backpropagation Under Angle-Parametrized Orthogonal Matrices](https://mlsanity.com/paper/2106.00003/) (2021)

## Abstract (arXiv)

Kernel normalization methods have been employed to improve robustness of optimization methods to reparametrization of convolution kernels, covariate shift, and to accelerate training of Convolutional Neural Networks (CNNs). However, our understanding of theoretical properties of these methods has lagged behind their success in applications. We develop a geometric framework to elucidate underlying mechanisms of a diverse range of kernel normalization methods. Our framework enables us to expound and identify geometry of space of normalized kernels. We analyze and delineate how state-of-the-art kernel normalization methods affect the geometry of search spaces of the stochastic gradient descent (SGD) algorithms in CNNs. Following our theoretical results, we propose a SGD algorithm with assurance of almost sure convergence of the methods to a solution at single minimum of classification loss of CNNs. Experimental results show that the proposed method achieves state-of-the-art performance for major image classification benchmarks with CNNs.