mlsanityData to 7 Oct 2026

Differentiable Top-k Classification Learning

arXiv 2206.07290 ↗First posted 15 Jun 2022cs.LG cs.CVcited by 21

Not on the map. Why?

Abstract (arXiv)

The top-k classification accuracy is one of the core metrics in machine learning. Here, k is conventionally a positive integer, such as 1 or 5, leading to top-1 or top-5 training objectives. In this work, we relax this assumption and optimize the model for multiple k simultaneously instead of using a single k. Leveraging recent advances in differentiable sorting and ranking, we propose a differentiable top-k cross-entropy classification loss. This allows training the network while not only considering the top-1 prediction, but also, e.g., the top-2 and top-5 predictions. We evaluate the proposed loss function for fine-tuning on state-of-the-art architectures, as well as for training from scratch. We find that relaxing k does not only produce better top-5 accuracies, but also leads to top-1 accuracy improvements. When fine-tuning publicly available ImageNet models, we achieve a new state-of-the-art for these models.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 15 Jun 2022.

Builds on

33 of 43 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  7. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  8. 2021Scaling Vision Transformerscited by 993
  9. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  10. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  11. 2021ImageNet-21K Pretraining for the Massescited by 607
  12. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
21 more
  1. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  2. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  3. 2020Meta Pseudo Labelscited by 372
  4. 2020Fast Differentiable Sorting and Rankingcited by 196
  5. 2019Differentiable Ranks and Sorting using Optimal Transportcited by 113
  6. 2019Stochastic Optimization of Sorting Networks via Continuous Relaxationscited by 111
  7. 2020Learning with Differentiable Perturbed Optimizerscited by 81
  8. 2021Differentiable Patch Selection for Image Recognitioncited by 68
  9. 2020SoftSort: A Continuous Relaxation for the argsort Operatorcited by 66
  10. 2017Learning with Average Top-k Losscited by 52
  11. 2018Smooth Loss Functions for Deep Top-k Classificationcited by 46
  12. 2015Loss Functions for Top-k Error: Analysis and Insightscited by 38
  13. 2019On the Consistency of Top-k Surrogate Lossescited by 35
  14. 2021Differentiable Sorting Networks for Scalable Sorting and Ranking Supervisioncited by 31
  15. 2015Top-k Multiclass SVMcited by 28
  16. 2021Set-valued classification -- overview via a unified frameworkcited by 27
  17. 2022Monotonic Differentiable Sorting Networkscited by 25
  18. 2021Learning with Algorithmic Supervision via Continuous Relaxationscited by 21
  19. 2017A Continuous Relaxation of Beam Search for End-to-end Training of Neural Sequence Modelscited by 10
  20. 2020Successive Halving Top-k Operatorcited by 2
  21. 2015Smaller Selection Networks for Cardinality Constraints Encodingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 2; 2025: 7; 2026: 3.

21 papers.

Sort:

By area:

  1. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 89
  2. 2023Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspectivecited by 26
  3. 2023Learning by Sorting: Self-supervised Learning with Group Ordering Constraintscited by 13
  4. 2025LapSum -- One Method to Differentiate Them All: Ranking, Sorting and Top-k Selectioncited by 12
  5. 2023Dual-Encoders for Extreme Multi-Label Classificationcited by 7
  6. 2023Holistic chemical evaluation reveals pitfalls in reaction prediction modelscited by 4
  7. 2025Context Guided Transformer Entropy Modeling for Video Compressioncited by 3
  8. 2024KDk: A Defense Mechanism Against Label Inference Attacks in Vertical Federated Learningcited by 2
  9. 2025Welfare as a Guiding Principle for Machine Learning -- From Compass, to Lens, to Roadmapcited by 2
  10. 2022Provable Robust Saliency-based Explanationscited by 1
  11. 2023Learning to Summarize Videos by Contrasting Clipscited by 1
  12. 2024Optimizing Predictive AI in Physical Design Flows with Mini Pixel Batch Gradient Descentcited by 1
9 more
  1. 2025MOCLIP: A Foundation Model for Large-Scale Nanophotonic Inverse Designcited by 1
  2. 2026Differentiable Knapsack and Top-k Operators via Dynamic Programmingcited by 1
  3. 2023Evaluation Metrics for DNNs Compressioncited by 0
  4. 2023Self-supervised Learning for Anomaly Detection in Computational Workflowscited by 0
  5. 2025Analyzing Transformer Models and Knowledge Distillation Approaches for Image Captioning on Edge AIcited by 0
  6. 2025Scalable and Precise Patch Robustness Certification for Deep Learning Models with Top-k Predictionscited by 0
  7. 2025Relative-Based Scaling Law for Neural Language Modelscited by 0
  8. 2026TT-Sparse: Learning Sparse Rule Models with Differentiable Truth Tablescited by 0
  9. 2026Learning Permutation from Structure Without Supervisioncited by 0