# OL'E: Orthogonal Low-rank Embedding, A Plug and Play Geometric Loss for Deep Learning

arXiv: https://arxiv.org/abs/1712.01727 · HTML: https://mlsanity.com/paper/1712.01727/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2017.

## Built on by (most cited first, 30 of 37)

- [FedBABU: Towards Enhanced Representation for Federated Image Classification](https://mlsanity.com/paper/2106.06042/) (2021)
- [Orthogonal Projection Loss](https://mlsanity.com/paper/2103.14021/) (2021)
- [The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy](https://mlsanity.com/paper/2203.06345/) (2022)
- [Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations](https://mlsanity.com/paper/2303.03307/) (2023)
- [Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Data](https://mlsanity.com/paper/2107.00648/) (2021)
- [Semi-orthogonal Embedding for Efficient Unsupervised Anomaly Segmentation](https://mlsanity.com/paper/2105.14737/) (2021)
- [Survival Prediction of Brain Cancer with Incomplete Radiology, Pathology, Genomics, and Demographic Data](https://mlsanity.com/paper/2203.04419/) (2022)
- [SphereFed: Hyperspherical Federated Learning](https://mlsanity.com/paper/2207.09413/) (2022)
- [PaCE: Parsimonious Concept Engineering for Large Language Models](https://mlsanity.com/paper/2406.04331/) (2024)
- [Parametric Scattering Networks](https://mlsanity.com/paper/2107.09539/) (2021)
- [Tune It or Don't Use It: Benchmarking Data-Efficient Image Classification](https://mlsanity.com/paper/2108.13122/) (2021)
- [Not All Frame Features Are Equal: Video-to-4D Generation via Decoupling Dynamic-Static Features](https://mlsanity.com/paper/2502.08377/) (2025)
- [Exploring a Principled Framework for Deep Subspace Clustering](https://mlsanity.com/paper/2503.17288/) (2025)
- [Genetic Programming-Based Evolutionary Deep Learning for Data-Efficient Image Classification](https://mlsanity.com/paper/2209.13233/) (2022)
- [Image Classification with Small Datasets: Overview and Benchmark](https://mlsanity.com/paper/2212.12478/) (2022)
- [Distortion-Disentangled Contrastive Learning](https://mlsanity.com/paper/2303.05066/) (2023)
- [Stabilizing Self-Consuming Diffusion Models with Latent Space Filtering](https://mlsanity.com/paper/2511.12742/) (2025)
- [Uncovering the Latent Potential of Deep Intermediate Representations](https://mlsanity.com/paper/2605.23033/) (2026)
- [Object-centric and memory-guided normality reconstruction for video anomaly detection](https://mlsanity.com/paper/2203.03677/) (2022)
- [Mutual Contrastive Low-rank Learning to Disentangle Whole Slide Image Representations for Glioma Grading](https://mlsanity.com/paper/2203.04013/) (2022)
- [Label Structure Preserving Contrastive Embedding for Multi-Label Learning with Missing Labels](https://mlsanity.com/paper/2209.01314/) (2022)
- [FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learning](https://mlsanity.com/paper/2406.02355/) (2024)
- [Geometric Understanding of Discriminability and Transferability for Visual Domain Adaptation](https://mlsanity.com/paper/2407.09524/) (2024)
- [Memory Wrap: a Data-Efficient and Interpretable Extension to Image Classification Models](https://mlsanity.com/paper/2106.01440/) (2021)
- [No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasets](https://mlsanity.com/paper/2309.01694/) (2023)
- [Enhancing Fine-grained Object Detection in Aerial Images via Orthogonal Mapping](https://mlsanity.com/paper/2407.17738/) (2024)
- [Learning to Balance: Diverse Normalization for Cloth-Changing Person Re-Identification](https://mlsanity.com/paper/2410.03977/) (2024)
- [ESS-ReduNet: Enhancing Subspace Separability of ReduNet via Dynamic Expansion with Bayesian Inference](https://mlsanity.com/paper/2411.17961/) (2024)
- [OAAE: Adversarial Autoencoders for Novelty Detection in Multi-modal Normality Case via Orthogonalized Latent Space](https://mlsanity.com/paper/2101.02358/) (2021)
- [Geometry-Aware Unsupervised Domain Adaptation](https://mlsanity.com/paper/2112.11041/) (2021)

## Abstract (arXiv)

Deep neural networks trained using a softmax layer at the top and the cross-entropy loss are ubiquitous tools for image classification. Yet, this does not naturally enforce intra-class similarity nor inter-class margin of the learned deep representations. To simultaneously achieve these two goals, different solutions have been proposed in the literature, such as the pairwise or triplet losses. However, such solutions carry the extra task of selecting pairs or triplets, and the extra computational burden of computing and learning for many combinations of them. In this paper, we propose a plug-and-play loss term for deep networks that explicitly reduces intra-class variance and enforces inter-class margin simultaneously, in a simple and elegant geometric manner. For each class, the deep features are collapsed into a learned linear subspace, or union of them, and inter-class subspaces are pushed to be as orthogonal as possible. Our proposed Orthogonal Low-rank Embedding (OL'E) does not require carefully crafting pairs or triplets of samples for training, and works standalone as a classification loss, being the first reported deep metric learning framework of its kind. Because of the improved margin between features of different classes, the resulting deep networks generalize better, are more discriminative, and more robust. We demonstrate improved classification performance in general object recognition, plugging the proposed loss term into existing off-the-shelf architectures. In particular, we show the advantage of the proposed loss in the small data/model scenario, and we significantly advance the state-of-the-art on the Stanford STL-10 benchmark.