mlsanityData to 7 Oct 2026

ELM: Embedding and Logit Margins for Long-Tail Learning

arXiv 2204.13208 ↗First posted 27 Apr 2022cs.LG stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

Long-tail learning is the problem of learning under skewed label distributions, which pose a challenge for standard learners. Several recent approaches for the problem have proposed enforcing a suitable margin in logit space. Such techniques are intuitive analogues of the guiding principle behind SVMs, and are equally applicable to linear models and neural models. However, when applied to neural models, such techniques do not explicitly control the geometry of the learned embeddings. This can be potentially sub-optimal, since embeddings for tail classes may be diffuse, resulting in poor generalization for these classes. We present Embedding and Logit Margins (ELM), a unified approach to enforce margins in logit space, and regularize the distribution of embeddings. This connects losses for long-tail learning to proposals in the literature on metric embedding, and contrastive learning. We theoretically show that minimising the proposed ELM objective helps reduce the generalisation gap. The ELM method is shown to perform well empirically, and results in tighter tail class embeddings.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 45 of its 77 references that are in the map. First posted on arXiv 27 Apr 2022.

Builds on

45 of 77 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  7. 2020Supervised Contrastive Learningcited by 2,803
  8. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  10. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  11. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  12. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 806
33 more
  1. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  2. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  3. 2020Long-tail learning via logit adjustmentcited by 605
  4. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  5. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 510
  6. 2020Balanced Meta-Softmax for Long-Tailed Visual Recognitioncited by 397
  7. 2020Debiased Contrastive Learningcited by 354
  8. 2020Equalization Loss for Long-Tailed Object Recognitioncited by 285
  9. 2020Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effectcited by 256
  10. 2020Rethinking the Value of Labels for Improving Class-Imbalanced Learningcited by 216
  11. 2021Distribution Alignment: A Unified Framework for Long-tail Visual Recognitioncited by 198
  12. 2014Deep Learning Face Representation by Joint Identification-Verificationcited by 162
  13. 2020Seesaw Loss for Long-Tailed Instance Segmentationcited by 146
  14. 2020Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspectivecited by 143
  15. 2020Feature Space Augmentation for Long-Tailed Datacited by 136
  16. 2021Contrastive Learning based Hybrid Networks for Long-Tailed Image Classificationcited by 130
  17. 2020Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasetscited by 122
  18. 2020Deep Representation Learning on Long-tailed Data: A Learnable Embedding Augmentation Perspectivecited by 120
  19. 2016Range Loss for Deep Face Recognition with Long-tailcited by 111
  20. 2017The Devil is in the Tails: Fine-grained Classification in the Wildcited by 89
  21. 2021Distributional Robustness Loss for Long-tail Learningcited by 84
  22. 2020Identifying and Compensating for Feature Deviation in Imbalanced Deep Learningcited by 76
  23. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  24. 2017Learning Spread-out Local Feature Descriptorscited by 34
  25. 2021PML: Progressive Margin Loss for Long-tailed Age Classificationcited by 32
  26. 2020Hybrid Discriminative-Generative Training via Contrastive Learningcited by 24
  27. 2021Procrustean Training for Imbalanced Deep Learningcited by 24
  28. 2016Reviving Threshold-Moving: a Simple Plug-in Bagging Ensemble for Binary and Multiclass Imbalanced Datacited by 23
  29. 2015Deep Linear Discriminant Analysiscited by 22
  30. 2019To Balance or Not to Balance: A Simple-yet-Effective Approach for Learning with Long-Tailed Distributionscited by 20
  31. 2018Efficient Training on Very Large Corpora via Gramian Estimationcited by 9
  32. 2020Mitigating Dataset Imbalance via Joint Generation and Classificationcited by 4
  33. 2020Improving Calibration in Deep Metric Learning With Cross-Example Softmaxcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 3; 2025: 0; 2026: 1.

10 papers.

  1. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  2. 2024Probabilistic Contrastive Learning for Long-Tailed Visual Recognitioncited by 42
  3. 2023On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Datacited by 23
  4. 2023Distributionally Robust Post-hoc Classifiers under Prior Shiftscited by 16
  5. 2024A Survey of Deep Long-Tail Classification Advancementscited by 11
  6. 2023Uncertainty-guided Boundary Learning for Imbalanced Social Event Detectioncited by 8
  7. 2023Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batchingcited by 5
  8. 2024Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Featurescited by 5
  9. 2023Long-Tail Learning with Rebalanced Contrastive Losscited by 3
  10. 2026Reducing Class-Wise Performance Disparity via Margin Regularizationcited by 0