mlsanityData to 14 Oct 2026

Disentangling Sampling and Labeling Bias for Learning in Large-Output Spaces

arXiv 2105.05736 ↗First posted 12 May 2021cs.LG stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

Negative sampling schemes enable efficient training given a large number of classes, by offering a means to approximate a computationally expensive loss function that takes all labels into account. In this paper, we present a new connection between these schemes and loss modification techniques for countering label imbalance. We show that different negative sampling schemes implicitly trade-off performance on dominant versus rare labels. Further, we provide a unified means to explicitly tackle both sampling bias, arising from working with a subset of all labels, and labeling bias, which is inherent to the data due to label imbalance. We empirically verify our findings on long-tail classification and retrieval benchmarks.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 21 of its 45 references that are in the map. First posted on arXiv 12 May 2021.

Builds on

21 of 45 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  4. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  5. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
  6. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  7. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 809
  8. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 705
  9. 2015Session-based Recommendations with Recurrent Neural Networkscited by 695
  10. 2020Long-tail learning via logit adjustmentcited by 618
  11. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 513
  12. 2020Equalization Loss for Long-Tailed Object Recognitioncited by 285
9 more
  1. 2017The Devil is in the Tails: Fine-grained Classification in the Wildcited by 89
  2. 2015Variance Reduction in SGD by Distributed Importance Samplingcited by 79
  3. 2017On Sampling Strategies for Neural Network-based Collaborative Filteringcited by 71
  4. 2018A no-regret generalization of hierarchical softmax to extreme multi-label classificationcited by 52
  5. 2017Adaptive Sampled Softmax with Kernel Based Samplingcited by 38
  6. 2019Sampled Softmax with Random Fourier Featurescited by 33
  7. 2018Stochastic Negative Mining for Learning with Large Output Spacescited by 29
  8. 2020Extreme Classification via Adversarial Softmax Approximationcited by 10
  9. 2018Augment and Reduce: Stochastic Inference for Large Categorical Distributionscited by 3

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 1; 2023: 4; 2024: 2.

9 papers.

  1. 2022NGAME: Negative Mining-aware Mini-batching for Extreme Classificationcited by 32
  2. 2021A data-centric approach for improving ambiguous labels with combined semi-supervised classification and clusteringcited by 9
  3. 2023How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Rankingcited by 8
  4. 2021General Greedy De-bias Learningcited by 7
  5. 2023Towards Memory-Efficient Training for Extremely Large Output Spaces -- Learning with 500k Labels on a Single Commodity GPUcited by 6
  6. 2023Beyond Accuracy: Statistical Measures and Benchmark for Evaluation of Representation from Self-Supervised Learningcited by 3
  7. 2023Importance of negative sampling in weak label learningcited by 1
  8. 2024Graph Regularized Encoder Training for Extreme Classificationcited by 1
  9. 2024On the Necessity of World Knowledge for Mitigating Missing Labels in Extreme Classificationcited by 0