mlsanityData to 7 Oct 2026

Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?

arXiv 1810.09102 ↗First posted 22 Oct 2018cs.LG cs.CV stat.MLcited by 62

Not on the map. Why?

Abstract (arXiv)

This paper seeks to answer the question: as the (near-) orthogonality of weights is found to be a favorable property for training deep convolutional neural networks, how can we enforce it in more effective and easy-to-use ways? We develop novel orthogonality regularizations on training deep CNNs, utilizing various advanced analytical tools such as mutual coherence and restricted isometry property. These plug-and-play regularizations can be conveniently incorporated into training almost any CNN without extra hassle. We then benchmark their effects on state-of-the-art models: ResNet, WideResNet, and ResNeXt, on several most popular computer vision datasets: CIFAR-10, CIFAR-100, SVHN and ImageNet. We observe consistent performance gains after applying those proposed regularizations, in terms of both the final accuracies achieved, and faster and more stable convergences. We have made our codes and pre-trained models publicly available: https://github.com/nbansal90/Can-we-Gain-More-from-Orthogonality.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 17; 2023: 6; 2024: 3; 2025: 5; 2026: 6.

62 papers.

Sort:

By area:

  1. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  2. 2021Self-Domain Adaptation for Face Anti-Spoofingcited by 49
  3. 2022CHEX: CHannel EXploration for CNN Model Compressioncited by 36
  4. 2021Fast and accurate optimization on the orthogonal manifold without retractioncited by 35
  5. 2022The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancycited by 35
  6. 2022projUNN: efficient method for training deep networks with unitary matricescited by 27
  7. 2021Estimating and Improving Fairness with Adversarial Learningcited by 26
  8. 2022Orthogonalising gradients to speed up neural network optimisationcited by 25
  9. 2021Taxonomy of Machine Learning Safety: A Survey and Primercited by 22
  10. 2023A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifoldcited by 21
  11. 2021Orthogonal Graph Neural Networkscited by 19
  12. 2024Lightweight Frequency Masker for Cross-Domain Few-Shot Semantic Segmentationcited by 17
38 more
  1. 2021Existence, Stability and Scalability of Orthogonal Convolutional Neural Networkscited by 16
  2. 2021Convolutional Normalization: Improving Deep Convolutional Network Robustness and Trainingcited by 15
  3. 2022Improving GANs for Long-Tailed Data through Group Spectral Regularizationcited by 14
  4. 2021Nonlinear Transform Induced Tensor Nuclear Norm for Tensor Completioncited by 13
  5. 2023Orthogonal Directions Constrained Gradient Method: from non-linear equality constraints to Stiefel manifoldcited by 13
  6. 2021Dual Metric Learning for Effective and Efficient Cross-Domain Recommendationscited by 12
  7. 2022Riemannian Hamiltonian methods for min-max optimization on manifoldscited by 12
  8. 2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 9
  9. 2022Batch Normalization Explainedcited by 8
  10. 2021An Orthogonal Classifier for Improving the Adversarial Robustness of Neural Networkscited by 7
  11. 2025ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specializationcited by 6
  12. 2021Coordinate descent on the orthogonal group for recurrent neural network trainingcited by 5
  13. 2022Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNscited by 5
  14. 2021Interpreting the Latent Space of Generative Adversarial Networks using Supervised Learningcited by 4
  15. 2021Encouraging Intra-Class Diversity Through a Reverse Contrastive Loss for Better Single-Source Domain Generalizationcited by 4
  16. 2021Better Training using Weight-Constrained Stochastic Dynamicscited by 4
  17. 2022Entangled Residual Mappingscited by 4
  18. 2022Improving Covariance Conditioning of the SVD Meta-layer by Orthogonalitycited by 4
  19. 2023Distortion-Disentangled Contrastive Learningcited by 4
  20. 2021Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networkscited by 3
  21. 2021Ortho-Shot: Low Displacement Rank Regularization with Data Augmentation for Few-Shot Learningcited by 3
  22. 2021Exploiting a Zoo of Checkpoints for Unseen Taskscited by 3
  23. 2022What can we Learn by Predicting Accuracy?cited by 3
  24. 2023Look Beyond Bias with Entropic Adversarial Data Augmentationcited by 3
  25. 2023Towards Better Orthogonality Regularization with Disentangled Norm in Training Deep CNNscited by 3
  26. 2025Residual Vector Quantization For Communication-Efficient Multi-Agent Perceptioncited by 3
  27. 2022Semantics-Depth-Symbiosis: Deeply Coupled Semi-Supervised Learning of Semantics and Depthcited by 2
  28. 2022Revisiting Architecture-aware Knowledge Distillation: Smaller Models and Faster Searchcited by 2
  29. 2024Learning in Wilson-Cowan model for metapopulationcited by 2
  30. 2021Quadric Hypersurface Intersection for Manifold Learning in Feature Spacecited by 1
  31. 2021Deep Learning meets Liveness Detection: Recent Advancements and Challengescited by 1
  32. 2022Learning Decoupling Features Through Orthogonality Regularizationcited by 1
  33. 2022Riemannian Stochastic Gradient Method for Nested Composition Optimizationcited by 1
  34. 2022On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networkscited by 1
  35. 2025Deep Disentangled Representation Network for Treatment Effect Estimationcited by 1
  36. 2026OrthoGeoLoRA: Geometric Parameter-Efficient Fine-Tuning for Structured Social Science Concept Retrieval on theWebcited by 1
  37. 2026Regularized Latent Dynamics Prediction is a Strong Baseline For Behavioral Foundation Modelscited by 1
  38. 2026Learning Hierarchical Orthogonal Prototypes for Generalized Few-Shot 3D Point Cloud Segmentationcited by 1

50 most cited of 62 All 62 (JSON)