mlsanityData to 7 Oct 2026

Deep Isometric Learning for Visual Recognition

arXiv 2006.16992 ↗First posted 30 Jun 2020cs.CVcited by 38

Not on the map. Why?

Abstract (arXiv)

Initialization, normalization, and skip connections are believed to be three indispensable techniques for training very deep convolutional neural networks and obtaining state-of-the-art performance. This paper shows that deep vanilla ConvNets without normalization nor skip connections can also be trained to achieve surprisingly good performance on standard image recognition benchmarks. This is achieved by enforcing the convolution kernels to be near isometric during initialization and training, as well as by using a variant of ReLU that is shifted towards being isometric. Further experiments show that if combined with skip connections, such near isometric networks can achieve performances on par with (for ImageNet) and better than (for COCO) the standard ResNet, even without normalization at all. Our code is available at https://github.com/HaozhiQi/ISONet.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 10; 2023: 7; 2024: 3; 2025: 4; 2026: 2.

38 papers.

Sort:

By area:

  1. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  2. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  3. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  4. 2021Dirichlet Energy Constrained Learning for Deep Graph Neural Networkscited by 97
  5. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  6. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  7. 2021Learning with Hyperspherical Uniformitycited by 35
  8. 2021Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flowscited by 30
  9. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  10. 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 24
  11. 2021Existence, Stability and Scalability of Orthogonal Convolutional Neural Networkscited by 16
  12. 2021Convolutional Normalization: Improving Deep Convolutional Network Robustness and Trainingcited by 15
26 more
  1. 2023MIMONets: Multiple-Input-Multiple-Output Neural Networks Exploiting Computation in Superpositioncited by 14
  2. 2022Revisiting Sparse Convolutional Model for Visual Recognitioncited by 13
  3. 2023Orthogonal Directions Constrained Gradient Method: from non-linear equality constraints to Stiefel manifoldcited by 13
  4. 2022Old can be Gold: Better Gradient Flow can Make Vanilla-GCNs Great Againcited by 11
  5. 2024Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularizationcited by 9
  6. 2025An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architecturescited by 8
  7. 2021Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Frameworkcited by 7
  8. 2025Robustness of Nonlinear Representation Learningcited by 7
  9. 2021A Generalized Zero-Shot Quantization of Deep Convolutional Neural Networks via Learned Weights Statisticscited by 6
  10. 2022Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNscited by 5
  11. 2022Orthogonal SVD Covariance Conditioning and Latent Disentanglementcited by 5
  12. 2024OrthCaps: An Orthogonal CapsNet with Sparse Attention Routing and Pruningcited by 5
  13. 2025IDInit: A Universal and Stable Initialization Method for Neural Network Trainingcited by 5
  14. 2025Enhancing Certified Robustness via Block Reflector Orthogonal Layers and Logit Annealing Losscited by 5
  15. 2022Improving Covariance Conditioning of the SVD Meta-layer by Orthogonalitycited by 4
  16. 2023Perceptual Group Tokenizer: Building Perception with Iterative Groupingcited by 4
  17. 2023Towards Better Orthogonality Regularization with Disentangled Norm in Training Deep CNNscited by 3
  18. 2023Get the Best of Both Worlds: Improving Accuracy and Transferability by Grassmann Class Representationcited by 3
  19. 2021PyKale: Knowledge-Aware Machine Learning from Multiple Sources in Pythoncited by 2
  20. 2022Kernel Normalized Convolutional Networkscited by 2
  21. 2022Revisiting Architecture-aware Knowledge Distillation: Smaller Models and Faster Searchcited by 2
  22. 2023On Interpretable Approaches to Cluster, Classify and Represent Multi-Subspace Data via Minimum Lossy Coding Length based on Rate-Distortion Theorycited by 0
  23. 2023Manifold-Preserving Transformers are Effective for Short-Long Range Encodingcited by 0
  24. 2024Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularizationcited by 0
  25. 2026GeoMotionGPT: Geometry-Aligned Motion Understanding with Large Language Modelscited by 0
  26. 2026Orthogonium : A Unified, Efficient Library of Orthogonal and 1-Lipschitz Building Blockscited by 0