# Role of Orthogonality Constraints in Improving Properties of Deep Networks for Image Classification

arXiv: https://arxiv.org/abs/2009.10762 · HTML: https://mlsanity.com/paper/2009.10762/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2020.

## Built on by (most cited first, 5 of 5)

- [Latent Space Regularization for Unsupervised Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/2104.02633/) (2021)
- [Road Scenes Segmentation Across Different Domains by Disentangling Latent Representations](https://mlsanity.com/paper/2108.03021/) (2021)
- [Interpretable COVID-19 Chest X-Ray Classification via Orthogonality Constraint](https://mlsanity.com/paper/2102.08360/) (2021)
- [Topological Persistence Guided Knowledge Distillation for Wearable Sensor Data](https://mlsanity.com/paper/2407.05315/) (2024)
- [Trustworthy AI in Digital Health: A Comprehensive Review of Robustness and Explainability](https://mlsanity.com/paper/2608.02238/) (2026)

## Abstract (arXiv)

Standard deep learning models that employ the categorical cross-entropy loss are known to perform well at image classification tasks. However, many standard models thus obtained often exhibit issues like feature redundancy, low interpretability, and poor calibration. A body of recent work has emerged that has tried addressing some of these challenges by proposing the use of new regularization functions in addition to the cross-entropy loss. In this paper, we present some surprising findings that emerge from exploring the role of simple orthogonality constraints as a means of imposing physics-motivated constraints common in imaging. We propose an Orthogonal Sphere (OS) regularizer that emerges from physics-based latent-representations under simplifying assumptions. Under further simplifying assumptions, the OS constraint can be written in closed-form as a simple orthonormality term and be used along with the cross-entropy loss function. The findings indicate that orthonormality loss function results in a) rich and diverse feature representations, b) robustness to feature sub-selection, c) better semantic localization in the class activation maps, and d) reduction in model calibration error. We demonstrate the effectiveness of the proposed OS regularization by providing quantitative and qualitative results on four benchmark datasets - CIFAR10, CIFAR100, SVHN and tiny ImageNet.