# Regularizing Neural Networks via Minimizing Hyperspherical Energy

arXiv: https://arxiv.org/abs/1906.04892 · HTML: https://mlsanity.com/paper/1906.04892/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2019.

## Built on by (most cited first, 25 of 25)

- [Controlling Text-to-Image Diffusion by Orthogonal Finetuning](https://mlsanity.com/paper/2306.07280/) (2023)
- [SphereFace2: Binary Classification is All You Need for Deep Face Recognition](https://mlsanity.com/paper/2108.01513/) (2021)
- [Learning with Hyperspherical Uniformity](https://mlsanity.com/paper/2103.01649/) (2021)
- [The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy](https://mlsanity.com/paper/2203.06345/) (2022)
- [SphereFace Revived: Unifying Hyperspherical Face Recognition](https://mlsanity.com/paper/2109.05565/) (2021)
- [Maximum Class Separation as Inductive Bias in One Matrix](https://mlsanity.com/paper/2206.08704/) (2022)
- [PoLAR: Polar-Decomposed Low-Rank Adapter Representation](https://mlsanity.com/paper/2506.03133/) (2025)
- [Hyperspherical Consistency Regularization](https://mlsanity.com/paper/2206.00845/) (2022)
- [Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap](https://mlsanity.com/paper/2303.06484/) (2023)
- [Continual Learning by Modeling Intra-Class Variation](https://mlsanity.com/paper/2210.05398/) (2022)
- [Iterative Teaching by Label Synthesis](https://mlsanity.com/paper/2110.14432/) (2021)
- [Spectral Scaling Laws in Language Models: How Effectively Do Feed-Forward Networks Use Their Latent Space?](https://mlsanity.com/paper/2510.00537/) (2025)
- [Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Losses](https://mlsanity.com/paper/2405.18045/) (2024)
- [Orthogonal Finetuning Made Scalable](https://mlsanity.com/paper/2506.19847/) (2025)
- [Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation](https://mlsanity.com/paper/2605.12492/) (2026)
- [Hyperspherically Regularized Networks for Self-Supervision](https://mlsanity.com/paper/2105.00925/) (2021)
- [Data-Efficient Learning via Minimizing Hyperspherical Energy](https://mlsanity.com/paper/2206.15204/) (2022)
- [HOFT: Householder Orthogonal Fine-tuning](https://mlsanity.com/paper/2505.16531/) (2025)
- [How to Use Graph Data in the Wild to Help Graph Anomaly Detection?](https://mlsanity.com/paper/2506.04190/) (2025)
- [Deep Simplex Classifier for Maximizing the Margin in Both Euclidean and Angular Spaces](https://mlsanity.com/paper/2212.11747/) (2022)
- [TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability](https://mlsanity.com/paper/2405.17678/) (2024)
- [Eccentric Regularization: Minimizing Hyperspherical Energy without explicit projection](https://mlsanity.com/paper/2104.11610/) (2021)
- [Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization](https://mlsanity.com/paper/2407.08374/) (2024)
- [Heterogeneous Federated Learning with Prototype Alignment and Upscaling](https://mlsanity.com/paper/2507.04310/) (2025)
- [Torus embeddings](https://mlsanity.com/paper/2603.03135/) (2026)

## Abstract (arXiv)

Inspired by the Thomson problem in physics where the distribution of multiple propelling electrons on a unit sphere can be modeled via minimizing some potential energy, hyperspherical energy minimization has demonstrated its potential in regularizing neural networks and improving their generalization power. In this paper, we first study the important role that hyperspherical energy plays in neural network training by analyzing its training dynamics. Then we show that naively minimizing hyperspherical energy suffers from some difficulties due to highly non-linear and non-convex optimization as the space dimensionality becomes higher, therefore limiting the potential to further improve the generalization. To address these problems, we propose the compressive minimum hyperspherical energy (CoMHE) as a more effective regularization for neural networks. Specifically, CoMHE utilizes projection mappings to reduce the dimensionality of neurons and minimizes their hyperspherical energy. According to different designs for the projection mapping, we propose several distinct yet well-performing variants and provide some theoretical guarantees to justify their effectiveness. Our experiments show that CoMHE consistently outperforms existing regularization methods, and can be easily applied to different neural networks.