mlsanityData to 7 Oct 2026

Orthogonal Convolutional Neural Networks

arXiv 1911.12207 ↗First posted 27 Nov 2019cs.CVcited by 122

Not on the map. Why?

Abstract (arXiv)

Deep convolutional neural networks are hindered by training instability and feature redundancy towards further performance improvement. A promising solution is to impose orthogonality on convolutional filters. We develop an efficient approach to impose filter orthogonality on a convolutional layer based on the doubly block-Toeplitz matrix representation of the convolutional kernel instead of using the common kernel orthogonality approach, which we show is only necessary but not sufficient for ensuring orthogonal convolutions. Our proposed orthogonal convolution requires no additional parameters and little computational overhead. This method consistently outperforms the kernel orthogonality alternative on a wide range of tasks such as image classification and inpainting under supervised, semi-supervised and unsupervised settings. Further, it learns more diverse and expressive features with better training stability, robustness, and generalization. Our code is publicly available at https://github.com/samaonline/Orthogonal-Convolutional-Neural-Networks.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 24; 2022: 28; 2023: 23; 2024: 17; 2025: 23; 2026: 7.

122 papers.

Sort:

By area:

  1. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 59
  2. 2022MogaNet: Multi-order Gated Aggregation Networkcited by 59
  3. 2021Orthogonal Projection Losscited by 44
  4. 2022Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networkscited by 39
  5. 2023Geometric Clifford Algebra Networkscited by 37
  6. 2021Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generationcited by 35
  7. 2022Systematic Literature Review: Quantum Machine Learning and its applicationscited by 32
  8. 2022LOT: Layer-wise Orthogonal Training on Improving ℓ2 Certified Robustnesscited by 32
  9. 2021Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networkscited by 30
  10. 2022projUNN: efficient method for training deep networks with unitary matricescited by 27
  11. 2022Orthogonalising gradients to speed up neural network optimisationcited by 25
  12. 2022Maximum Class Separation as Inductive Bias in One Matrixcited by 21
38 more
  1. 2023A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifoldcited by 21
  2. 2021Orthogonal Graph Neural Networkscited by 19
  3. 2022SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Mapscited by 19
  4. 2023Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iterationcited by 19
  5. 2023Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraintscited by 18
  6. 2021An Information Theory-inspired Strategy for Automatic Network Pruningcited by 17
  7. 2021Orthogonal Ensemble Networks for Biomedical Image Segmentationcited by 16
  8. 2021Existence, Stability and Scalability of Orthogonal Convolutional Neural Networkscited by 16
  9. 2022Trainability Preserving Neural Pruningcited by 16
  10. 2021Convolutional Normalization: Improving Deep Convolutional Network Robustness and Trainingcited by 15
  11. 2022RetrievalGuard: Provably Robust 1-Nearest Neighbor Image Retrievalcited by 15
  12. 2023Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gapcited by 14
  13. 2021Nonlinear Transform Induced Tensor Nuclear Norm for Tensor Completioncited by 13
  14. 2022Federated Learning of Large Models at the Edge via Principal Sub-Model Trainingcited by 13
  15. 2022Riemannian Hamiltonian methods for min-max optimization on manifoldscited by 12
  16. 2025Prior-guided Hierarchical Harmonization Network for Efficient Image Dehazingcited by 12
  17. 2026Mano: Restriking Manifold Optimization for LLM Trainingcited by 12
  18. 2021Spectral Leakage and Rethinking the Kernel Size in CNNscited by 11
  19. 2024ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networkscited by 11
  20. 2022Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transportcited by 10
  21. 2022Dynamical systems' based neural networkscited by 10
  22. 2023Robust One-Class Classification with Signed Distance Function using 1-Lipschitz Neural Networkscited by 10
  23. 2024Enhancing Feature Diversity Boosts Channel-Adaptive Vision Transformerscited by 10
  24. 2025Riemannian Optimization for LoRA on the Stiefel Manifoldcited by 10
  25. 2022O-ViT: Orthogonal Vision Transformercited by 9
  26. 2023Koopman-based generalization bound: New aspect for full-rank weightscited by 9
  27. 2024Efficient Pareto Manifold Learning with Low-Rank Structurecited by 9
  28. 2024Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularizationcited by 9
  29. 2025An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architecturescited by 8
  30. 2021An Orthogonal Classifier for Improving the Adversarial Robustness of Neural Networkscited by 7
  31. 2021Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Frameworkcited by 7
  32. 2023Decentralized Riemannian Algorithm for Nonconvex Minimax Problemscited by 7
  33. 2024Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layerscited by 7
  34. 2025Approximation theory for 1-Lipschitz ResNetscited by 7
  35. 2021Compact and Optimal Deep Learning with Recurrent Parameter Generatorscited by 5
  36. 2022Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNscited by 5
  37. 2022Orthogonal SVD Covariance Conditioning and Latent Disentanglementcited by 5
  38. 2023Deep Diversity-Enhanced Feature Representation of Hyperspectral Imagescited by 5

50 most cited of 122 All 122 (JSON)