# Improved Training Speed, Accuracy, and Data Utilization Through Loss Function Optimization

arXiv: https://arxiv.org/abs/1905.11528 · HTML: https://mlsanity.com/paper/1905.11528/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2019.

## Built on by (most cited first, 28 of 28)

- [PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions](https://mlsanity.com/paper/2204.12511/) (2022)
- [Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learning](https://mlsanity.com/paper/2206.03996/) (2022)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Loss Function Discovery for Object Detection via Convergence-Simulation Driven Search](https://mlsanity.com/paper/2102.04700/) (2021)
- [Searching for Robustness: Loss Learning for Noisy Classification Tasks](https://mlsanity.com/paper/2103.00243/) (2021)
- [Learning an Explicit Hyperparameter Prediction Function Conditioned on Tasks](https://mlsanity.com/paper/2107.02378/) (2021)
- [Learning Symbolic Model-Agnostic Loss Functions via Meta-Learning](https://mlsanity.com/paper/2209.08907/) (2022)
- [Efficient Activation Function Optimization through Surrogate Modeling](https://mlsanity.com/paper/2301.05785/) (2023)
- [Meta-Learning Adaptive Loss Functions](https://mlsanity.com/paper/2301.13247/) (2023)
- [Meta-tuning Loss Functions and Data Augmentation for Few-shot Object Detection](https://mlsanity.com/paper/2304.12161/) (2023)
- [Evolving GAN Formulations for Higher Quality Image Synthesis](https://mlsanity.com/paper/2102.08578/) (2021)
- [Problem Learning: Towards the Free Will of Machines](https://mlsanity.com/paper/2109.00177/) (2021)
- [Predicting O-GlcNAcylation Sites in Mammalian Proteins with Transformers and RNNs Trained with a New Loss Function](https://mlsanity.com/paper/2402.17131/) (2024)
- [Neural Loss Function Evolution for Large-Scale Image Classifier Convolutional Neural Networks](https://mlsanity.com/paper/2403.08793/) (2024)
- [Next Generation Loss Function for Image Classification](https://mlsanity.com/paper/2404.12948/) (2024)
- [Signature-Graph Networks](https://mlsanity.com/paper/2110.11551/) (2021)
- [GCCN: Global Context Convolutional Network](https://mlsanity.com/paper/2110.11664/) (2021)
- [Fast and Efficient Local Search for Genetic Programming Based Loss Function Learning](https://mlsanity.com/paper/2403.00865/) (2024)
- [GANetic Loss for Generative Adversarial Networks with a Focus on Medical Applications](https://mlsanity.com/paper/2406.05023/) (2024)
- [Evolving parametrized Loss for Image Classification Learning on Small Datasets](https://mlsanity.com/paper/2103.08249/) (2021)
- [Generating meta-learning tasks to evolve parametric loss for classification learning](https://mlsanity.com/paper/2111.10583/) (2021)
- [Evolving Loss Functions for Specific Image Augmentation Techniques](https://mlsanity.com/paper/2404.06633/) (2024)
- [Prediction of Stocks Index Price using Quantum GANs](https://mlsanity.com/paper/2509.12286/) (2025)
- [Regularisation in neural networks: a survey and empirical analysis of approaches](https://mlsanity.com/paper/2601.23131/) (2026)
- [Rethinking the Harmonic Loss via Non-Euclidean Distance Layers](https://mlsanity.com/paper/2603.10225/) (2026)
- [Distribution-Free Pretraining of Classification Losses via Evolutionary Dynamics](https://mlsanity.com/paper/2605.03722/) (2026)
- [Distributional Loss for Robust Classification](https://mlsanity.com/paper/2606.13223/) (2026)
- [EvoRank: LLM-Guided Evolution of Multi-Objective Learning-to-Rank Pipelines](https://mlsanity.com/paper/2609.22196/) (2026)

## Abstract (arXiv)

As the complexity of neural network models has grown, it has become increasingly important to optimize their design automatically through metalearning. Methods for discovering hyperparameters, topologies, and learning rate schedules have lead to significant increases in performance. This paper shows that loss functions can be optimized with metalearning as well, and result in similar improvements. The method, Genetic Loss-function Optimization (GLO), discovers loss functions de novo, and optimizes them for a target task. Leveraging techniques from genetic programming, GLO builds loss functions hierarchically from a set of operators and leaf nodes. These functions are repeatedly recombined and mutated to find an optimal structure, and then a covariance-matrix adaptation evolutionary strategy (CMA-ES) is used to find optimal coefficients. Networks trained with GLO loss functions are found to outperform the standard cross-entropy loss on standard image classification tasks. Training with these new loss functions requires fewer steps, results in lower test error, and allows for smaller datasets to be used. Loss-function optimization thus provides a new dimension of metalearning, and constitutes an important step towards AutoML.