# Learning Surrogate Losses

arXiv: https://arxiv.org/abs/1905.10108 · HTML: https://mlsanity.com/paper/1905.10108/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2019.

## Built on by (most cited first, 21 of 21)

- [Meta-learning PINN loss functions](https://mlsanity.com/paper/2107.05544/) (2021)
- [Gradient-based Bi-level Optimization for Deep Learning: A Survey](https://mlsanity.com/paper/2207.11719/) (2022)
- [sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel Classification](https://mlsanity.com/paper/2108.10566/) (2021)
- [MetricOpt: Learning to Optimize Black-Box Evaluation Metrics](https://mlsanity.com/paper/2104.10631/) (2021)
- [Learning Symbolic Model-Agnostic Loss Functions via Meta-Learning](https://mlsanity.com/paper/2209.08907/) (2022)
- [Amortized Synthesis of Constrained Configurations Using a Differentiable Surrogate](https://mlsanity.com/paper/2106.09019/) (2021)
- [Zero Grads: Learning Local Surrogate Losses for Non-Differentiable Graphics](https://mlsanity.com/paper/2308.05739/) (2023)
- [An Invitation to Deep Reinforcement Learning](https://mlsanity.com/paper/2312.08365/) (2023)
- [Perplexity Trap: PLM-Based Retrievers Overrate Low Perplexity Documents](https://mlsanity.com/paper/2503.08684/) (2025)
- [DL-DDA -- Deep Learning based Dynamic Difficulty Adjustment with UX and Gameplay constraints](https://mlsanity.com/paper/2106.03075/) (2021)
- [Distribution-aware Margin Calibration for Semantic Segmentation in Images](https://mlsanity.com/paper/2112.11554/) (2021)
- [Improve Noise Tolerance of Robust Loss via Noise-Awareness](https://mlsanity.com/paper/2301.07306/) (2023)
- [Relational Surrogate Loss Learning](https://mlsanity.com/paper/2202.13197/) (2022)
- [EXACT: How to Train Your Accuracy](https://mlsanity.com/paper/2205.09615/) (2022)
- [LegendreTron: Uprising Proper Multiclass Loss Learning](https://mlsanity.com/paper/2301.11695/) (2023)
- [Goal-Oriented Time-Series Forecasting: Foundation Framework Design](https://mlsanity.com/paper/2504.17493/) (2025)
- [Fast and Efficient Local Search for Genetic Programming Based Loss Function Learning](https://mlsanity.com/paper/2403.00865/) (2024)
- [Selective Test-Time Adaptation for Unsupervised Anomaly Detection using Neural Implicit Representations](https://mlsanity.com/paper/2410.03306/) (2024)
- [Stateless actor-critic for instance segmentation with high-level priors](https://mlsanity.com/paper/2107.02600/) (2021)
- [An Algebraic Approach to Learning and Grounding](https://mlsanity.com/paper/2204.02813/) (2022)
- [Robustly Invertible Nonlinear Dynamics and the BiLipREN: From Inversion-Based Control to Generative Trajectory Modelling](https://mlsanity.com/paper/2607.10026/) (2026)

## Abstract (arXiv)

The minimization of loss functions is the heart and soul of Machine Learning. In this paper, we propose an off-the-shelf optimization approach that can minimize virtually any non-differentiable and non-decomposable loss function (e.g. Miss-classification Rate, AUC, F1, Jaccard Index, Mathew Correlation Coefficient, etc.) seamlessly. Our strategy learns smooth relaxation versions of the true losses by approximating them through a surrogate neural network. The proposed loss networks are set-wise models which are invariant to the order of mini-batch instances. Ultimately, the surrogate losses are learned jointly with the prediction model via bilevel optimization. Empirical results on multiple datasets with diverse real-life loss functions compared with state-of-the-art baselines demonstrate the efficiency of learning surrogate losses.