# Training Deep Neural Networks via Direct Loss Minimization

arXiv: https://arxiv.org/abs/1511.06411 · HTML: https://mlsanity.com/paper/1511.06411/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2015.

## Built on by (most cited first, 25 of 25)

- [Implicit MLE: Backpropagating Through Discrete Exponential Family Distributions](https://mlsanity.com/paper/2106.01798/) (2021)
- [Theseus: A Library for Differentiable Nonlinear Optimization](https://mlsanity.com/paper/2207.09442/) (2022)
- [RankSim: Ranking Similarity Regularization for Deep Imbalanced Regression](https://mlsanity.com/paper/2205.15236/) (2022)
- [Tuning computer vision models with task rewards](https://mlsanity.com/paper/2302.08242/) (2023)
- [Energy-based learning algorithms for analog computing: a comparative study](https://mlsanity.com/paper/2312.15103/) (2023)
- [Stochastic Optimization of Areas Under Precision-Recall Curves with Provable Convergence](https://mlsanity.com/paper/2104.08736/) (2021)
- [Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation](https://mlsanity.com/paper/2205.03076/) (2022)
- [AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks](https://mlsanity.com/paper/2103.14026/) (2021)
- [Momentum Accelerates the Convergence of Stochastic AUPRC Maximization](https://mlsanity.com/paper/2107.01173/) (2021)
- [Adaptive Perturbation-Based Gradient Estimation for Discrete Latent Variable Models](https://mlsanity.com/paper/2209.04862/) (2022)
- [Training Over-parameterized Models with Non-decomposable Objectives](https://mlsanity.com/paper/2107.04641/) (2021)
- [MetricOpt: Learning to Optimize Black-Box Evaluation Metrics](https://mlsanity.com/paper/2104.10631/) (2021)
- [Not All Pairs are Equal: Hierarchical Learning for Average-Precision-Oriented Video Retrieval](https://mlsanity.com/paper/2407.15566/) (2024)
- [Exploring the Algorithm-Dependent Generalization of AUPRC Optimization with List Stability](https://mlsanity.com/paper/2209.13262/) (2022)
- [LPGD: A General Framework for Backpropagation through Embedded Optimization Layers](https://mlsanity.com/paper/2407.05920/) (2024)
- [An Invitation to Deep Reinforcement Learning](https://mlsanity.com/paper/2312.08365/) (2023)
- [A Surrogate Objective Framework for Prediction+Optimization with Soft Constraints](https://mlsanity.com/paper/2111.11358/) (2021)
- [Learning with Latent Structures in Natural Language Processing: A Survey](https://mlsanity.com/paper/2201.00490/) (2022)
- [Searching Parameterized AP Loss for Object Detection](https://mlsanity.com/paper/2112.05138/) (2021)
- [Ranking Regularization for Critical Rare Classes: Minimizing False Positives at a High True Positive Rate](https://mlsanity.com/paper/2304.00049/) (2023)
- [Stateless actor-critic for instance segmentation with high-level priors](https://mlsanity.com/paper/2107.02600/) (2021)
- [The Contextual Appointment Scheduling Problem](https://mlsanity.com/paper/2108.05531/) (2021)
- [Implicit Training of Energy Model for Structure Prediction](https://mlsanity.com/paper/2211.11649/) (2022)
- [Aligning Multiclass Neural Network Classifier Criterion with Task Performance Metrics](https://mlsanity.com/paper/2405.20954/) (2024)
- [On The Statistical Representation Properties Of The Perturb-Softmax And The Perturb-Argmax Probability Distributions](https://mlsanity.com/paper/2406.02180/) (2024)

## Abstract (arXiv)

Supervised training of deep neural nets typically relies on minimizing cross-entropy. However, in many domains, we are interested in performing well on metrics specific to the application. In this paper we propose a direct loss minimization approach to train deep neural networks, which provably minimizes the application-specific loss function. This is often non-trivial, since these functions are neither smooth nor decomposable and thus are not amenable to optimization with standard gradient-based methods. We demonstrate the effectiveness of our approach in the context of maximizing average precision for ranking problems. Towards this goal, we develop a novel dynamic programming algorithm that can efficiently compute the weight updates. Our approach proves superior to a variety of baselines in the context of action classification and object detection, especially in the presence of label noise.