# Learning with Average Top-k Loss

arXiv: https://arxiv.org/abs/1705.08826 · HTML: https://mlsanity.com/paper/1705.08826/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2017.

## Built on by (most cited first, 30 of 52)

- [Distributionally Robust Federated Averaging](https://mlsanity.com/paper/2102.12660/) (2021)
- [On Tilted Losses in Machine Learning: Theory and Applications](https://mlsanity.com/paper/2109.06141/) (2021)
- [TkML-AP: Adversarial Attacks to Top-k Multi-Label Learning](https://mlsanity.com/paper/2108.00146/) (2021)
- [Rank-based Decomposable Losses in Machine Learning: A Survey](https://mlsanity.com/paper/2207.08768/) (2022)
- [Sum of Ranked Range Loss for Supervised Learning](https://mlsanity.com/paper/2106.03300/) (2021)
- [Bayesian Nonparametric Submodular Video Partition for Robust Anomaly Detection](https://mlsanity.com/paper/2203.12840/) (2022)
- [TransFace++: Rethinking the Face Recognition Paradigm with a Focus on Accuracy, Efficiency, and Security](https://mlsanity.com/paper/2308.10133/) (2023)
- [Differentiable Top-k Classification Learning](https://mlsanity.com/paper/2206.07290/) (2022)
- [Distributionally Robust Models with Parametric Likelihood Ratios](https://mlsanity.com/paper/2204.06340/) (2022)
- [Large-scale Optimization of Partial AUC in a Range of False Positive Rates](https://mlsanity.com/paper/2203.01505/) (2022)
- [Balancing the Scales: A Theoretical and Algorithmic Framework for Learning from Imbalanced Data](https://mlsanity.com/paper/2502.10381/) (2025)
- [TopoFR: A Closer Look at Topology Alignment on Face Recognition](https://mlsanity.com/paper/2410.10587/) (2024)
- [A Survey of Deep Long-Tail Classification Advancements](https://mlsanity.com/paper/2404.15593/) (2024)
- [K-SAM: Sharpness-Aware Minimization at the Speed of SGD](https://mlsanity.com/paper/2210.12864/) (2022)
- [A Simple Yet Effective Strategy to Robustify the Meta Learning Paradigm](https://mlsanity.com/paper/2310.00708/) (2023)
- [Implicit Rate-Constrained Optimization of Non-decomposable Objectives](https://mlsanity.com/paper/2107.10960/) (2021)
- [Are Negative Samples Necessary in Entity Alignment? An Approach with High Performance, Scalability and Robustness](https://mlsanity.com/paper/2108.05278/) (2021)
- [MAPPING: Model Average with Post-processing for Stroke Lesion Segmentation](https://mlsanity.com/paper/2211.15486/) (2022)
- [Lower-Left Partial AUC: An Effective and Efficient Optimization Metric for Recommendation](https://mlsanity.com/paper/2403.00844/) (2024)
- [Improved Diversity-Promoting Collaborative Metric Learning for Recommendation](https://mlsanity.com/paper/2409.01012/) (2024)
- [Improved Balanced Classification with Theoretically Grounded Loss Functions](https://mlsanity.com/paper/2512.23947/) (2025)
- [Spectral risk-based learning using unbounded losses](https://mlsanity.com/paper/2105.04816/) (2021)
- [Adaptive Ranking-based Sample Selection for Weakly Supervised Class-imbalanced Text Classification](https://mlsanity.com/paper/2210.03092/) (2022)
- [Stochastic Optimization for Spectral Risk Measures](https://mlsanity.com/paper/2212.05149/) (2022)
- [An Efficient Stochastic Algorithm for Decentralized Nonconvex-Strongly-Concave Minimax Optimization](https://mlsanity.com/paper/2212.02387/) (2022)
- [How Important is Importance Sampling for Deep Budgeted Training?](https://mlsanity.com/paper/2110.14283/) (2021)
- [Asymptotically Unbiased Instance-wise Regularized Partial AUC Optimization: Theory and Algorithm](https://mlsanity.com/paper/2210.03967/) (2022)
- [A Survey of Learning Criteria Going Beyond the Usual Risk](https://mlsanity.com/paper/2110.04996/) (2021)
- [Class-Level Logit Perturbation](https://mlsanity.com/paper/2209.05668/) (2022)
- [Revisiting adversarial training for the worst-performing class](https://mlsanity.com/paper/2302.08872/) (2023)

## Abstract (arXiv)

In this work, we introduce the em average top-k (atk) loss as a new aggregate loss for supervised learning, which is the average over the k largest individual losses over a training dataset. We show that the atk loss is a natural generalization of the two widely used aggregate losses, namely the average loss and the maximum loss, but can combine their advantages and mitigate their drawbacks to better adapt to different data distributions. Furthermore, it remains a convex function over all individual losses, which can lead to convex optimization problems that can be solved effectively with conventional gradient-based methods. We provide an intuitive interpretation of the atk loss based on its equivalent effect on the continuous individual loss functions, suggesting that it can reduce the penalty on correctly classified data. We further give a learning theory analysis of matk learning on the classification calibration of the atk loss and the error bounds of atk-SVM. We demonstrate the applicability of minimum average top-k learning for binary classification and regression using synthetic and real datasets.