mlsanityData to 14 Oct 2026

Towards Memory-Efficient Training for Extremely Large Output Spaces -- Learning with 500k Labels on a Single Commodity GPU

arXiv 2306.03725 ↗First posted 6 Jun 2023cs.LG cs.AI cs.DCcited by 6

Not on the map. Why?

Abstract (arXiv)

In classification problems with large output spaces (up to millions of labels), the last layer can require an enormous amount of memory. Using sparse connectivity would drastically reduce the memory requirements, but as we show below, it can result in much diminished predictive performance of the model. Fortunately, we found that this can be mitigated by introducing a penultimate layer of intermediate size. We further demonstrate that one can constrain the connectivity of the sparse layer to be uniform, in the sense that each output neuron will have the exact same number of incoming connections. This allows for efficient implementations of sparse matrix multiplication and connection redistribution on GPU hardware. Via a custom CUDA implementation, we show that the proposed approach can scale to datasets with 670,000 labels on a single commodity GPU with only 4GB memory.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 18 of its 34 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

18 of 34 references.

References · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  3. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  4. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  5. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 213
  6. 2017Deep Rewiring: Training very sparse deep networkscited by 139
  7. 2018AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classificationcited by 102
  8. 2015Inferring Networks of Substitutable and Complementary Productscited by 92
  9. 2016DiSMEC - Distributed Sparse Machines for Extreme Multi-label Classificationcited by 71
  10. 2019Bonsai -- Diverse and Shallow Trees for Extreme Multi-label Classificationcited by 61
  11. 2021LightXML: Transformer with Dynamic Negative Sampling for High-Performance Extreme Multi-label Text Classificationcited by 61
  12. 2018A no-regret generalization of hierarchical softmax to extreme multi-label classificationcited by 52
6 more
  1. 2019Extreme Classification in Log Memory using Count-Min Sketch: A Case Study of Amazon Search with 50M Productscited by 36
  2. 2018Stochastic Negative Mining for Learning with Large Output Spacescited by 29
  3. 2022On Missing Labels, Long-tails and Propensities in Extreme Multi-label Classificationcited by 15
  4. 2022CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classificationcited by 12
  5. 2014Conditional Probability Tree Estimation Analysis and Algorithmscited by 10
  6. 2021Disentangling Sampling and Labeling Bias for Learning in Large-Output Spacescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 2.

6 papers.

  1. 2024Sparse maximal update parameterization: A holistic approach to sparse training dynamicscited by 16
  2. 2024Navigating Extremes: Dynamic Sparsity in Large Output Spacescited by 3
  3. 2025SD2: Self-Distilled Sparse Drafterscited by 1
  4. 2026HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spacescited by 1
  5. 2025NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensemblingcited by 0
  6. 2026Beyond Masked Sparsity: SNACK Enables Truly Sparse Neural Networks on GPUcited by 0