# The Z-loss: a shift and scale invariant classification loss belonging to the Spherical Family

arXiv: https://arxiv.org/abs/1604.08859 · HTML: https://mlsanity.com/paper/1604.08859/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2016.

## Built on by (most cited first, 12 of 12)

- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration](https://mlsanity.com/paper/2512.22382/) (2025)
- [YuLan-Mini: An Open Data-efficient Language Model](https://mlsanity.com/paper/2412.17743/) (2024)
- [Voxtral Realtime](https://mlsanity.com/paper/2602.11298/) (2026)
- [The Design Space of Tri-Modal Masked Diffusion Models](https://mlsanity.com/paper/2602.21472/) (2026)
- [Take One Step at a Time to Know Incremental Utility of Demonstration: An Analysis on Reranking for Few-Shot In-Context Learning](https://mlsanity.com/paper/2311.09619/) (2023)
- [Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks](https://mlsanity.com/paper/2606.29176/) (2026)
- [GROOT: Corrective Reward Optimization for Generative Sequential Labeling](https://mlsanity.com/paper/2209.14694/) (2022)
- [How Does Beam Search improve Span-Level Confidence Estimation in Generative Sequence Labeling?](https://mlsanity.com/paper/2212.10767/) (2022)
- [Dead Directions: Geometric Singular Learning](https://mlsanity.com/paper/2606.05957/) (2026)
- [Attention Projection Mixing with Exogenous Anchors](https://mlsanity.com/paper/2601.08131/) (2026)
- [Z-Loss Backward Geometry in Dense Output Heads and Sparse Routers](https://mlsanity.com/paper/2609.16179/) (2026)

## Abstract (arXiv)

Despite being the standard loss function to train multi-class neural networks, the log-softmax has two potential limitations. First, it involves computations that scale linearly with the number of output classes, which can restrict the size of problems we are able to tackle with current hardware. Second, it remains unclear how close it matches the task loss such as the top-k error rate or other non-differentiable evaluation metrics which we aim to optimize ultimately. In this paper, we introduce an alternative classification loss function, the Z-loss, which is designed to address these two issues. Unlike the log-softmax, it has the desirable property of belonging to the spherical loss family (Vincent et al., 2015), a class of loss functions for which training can be performed very efficiently with a complexity independent of the number of output classes. We show experimentally that it significantly outperforms the other spherical loss functions previously investigated. Furthermore, we show on a word language modeling task that it also outperforms the log-softmax with respect to certain ranking scores, such as top-k scores, suggesting that the Z-loss has the flexibility to better match the task loss. These qualities thus makes the Z-loss an appealing candidate to train very efficiently large output networks such as word-language models or other extreme classification problems. On the One Billion Word (Chelba et al., 2014) dataset, we are able to train a model with the Z-loss 40 times faster than the log-softmax and more than 4 times faster than the hierarchical softmax.