mlsanityData to 7 Oct 2026

The Tree Loss: Improving Generalization with Many Classes

arXiv 2204.07727 ↗First posted 16 Apr 2022cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Multi-class classification problems often have many semantically similar classes. For example, 90 of ImageNet's 1000 classes are for different breeds of dog. We should expect that these semantically similar classes will have similar parameter vectors, but the standard cross entropy loss does not enforce this constraint. We introduce the tree loss as a drop-in replacement for the cross entropy loss. The tree loss re-parameterizes the parameter matrix in order to guarantee that semantically similar classes will have similar parameter vectors. Using simple properties of stochastic gradient descent, we show that the tree loss's generalization error is asymptotically better than the cross entropy loss's. We then validate these theoretical results on synthetic data, image data (CIFAR100, ImageNet), and text data (Twitter).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 34 references that are in the map. First posted on arXiv 16 Apr 2022.

Builds on

16 of 34 references.

References · 16

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  9. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  10. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  11. 2019Making Better Mistakes: Leveraging Class Hierarchies with Deep Networkscited by 112
  12. 2015Loss Functions for Top-k Error: Analysis and Insightscited by 38
4 more
  1. 2016emoji2vec: Learning Emoji Representations from their Descriptioncited by 28
  2. 2017A hierarchical loss and its problems when classifying non-hierarchicallycited by 15
  3. 2018Effectiveness of Hierarchical Softmax in Large Scale Classification Taskscited by 8
  4. 2020SimLoss: Class Similarities in Cross Entropycited by 5

Built on by

Not cited yet.