# Neural Loss Function Evolution for Large-Scale Image Classifier Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2403.08793 · HTML: https://mlsanity.com/paper/2403.08793/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Builds on (most cited first, 21 of 21)

- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [Neural Optimizer Search with Reinforcement Learning](https://mlsanity.com/paper/1709.07417/) (2017)
- [PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions](https://mlsanity.com/paper/2204.12511/) (2022)
- [Meta-Learning via Learned Loss](https://mlsanity.com/paper/1906.05374/) (2019)
- [Evolving Normalization-Activation Layers](https://mlsanity.com/paper/2004.02967/) (2020)
- [Improved Training Speed, Accuracy, and Data Utilization Through Loss Function Optimization](https://mlsanity.com/paper/1905.11528/) (2019)
- [Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark](https://mlsanity.com/paper/1806.01427/) (2018)
- [Loss Function Discovery for Object Detection via Convergence-Simulation Driven Search](https://mlsanity.com/paper/2102.04700/) (2021)
- [AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks](https://mlsanity.com/paper/2103.14026/) (2021)
- [Evolutionary Optimization of Deep Learning Activation Functions](https://mlsanity.com/paper/2002.07224/) (2020)
- [Searching for Robustness: Loss Learning for Noisy Classification Tasks](https://mlsanity.com/paper/2103.00243/) (2021)
- [Learning Symbolic Model-Agnostic Loss Functions via Meta-Learning](https://mlsanity.com/paper/2209.08907/) (2022)

## Built on by (most cited first, 2 of 2)

- [Evolving Loss Functions for Specific Image Augmentation Techniques](https://mlsanity.com/paper/2404.06633/) (2024)
- [Neural Optimizer Equation, Decay Function, and Learning Rate Schedule Joint Evolution](https://mlsanity.com/paper/2404.06679/) (2024)

## Abstract (arXiv)

For classification, neural networks typically learn by minimizing cross-entropy, but are evaluated and compared using accuracy. This disparity suggests neural loss function search (NLFS), the search for a drop-in replacement loss function of cross-entropy for neural networks. We apply NLFS to image classifier convolutional neural networks. We propose a new search space for NLFS that encourages more diverse loss functions to be explored, and a surrogate function that accurately transfers to large-scale convolutional neural networks. We search the space using regularized evolution, a mutation-only aging genetic algorithm. After evolution and a proposed loss function elimination protocol, we transferred the final loss functions across multiple architectures, datasets, and image augmentation techniques to assess generalization. In the end, we discovered three new loss functions, called NeuroLoss1, NeuroLoss2, and NeuroLoss3 that were able to outperform cross-entropy in terms of a higher mean test accuracy as a simple drop-in replacement loss function across the majority of experiments.