# Distribution-Free Pretraining of Classification Losses via Evolutionary Dynamics

arXiv: https://arxiv.org/abs/2605.03722 · HTML: https://mlsanity.com/paper/2605.03722/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 5 May 2026.

## Builds on (most cited first, 10 of 10)

- [Large-Margin Softmax Loss for Convolutional Neural Networks](https://mlsanity.com/paper/1612.02295/) (2016)
- [Additive Margin Softmax for Face Verification](https://mlsanity.com/paper/1801.05599/) (2018)
- [A General and Adaptive Robust Loss Function](https://mlsanity.com/paper/1701.03077/) (2017)
- [Time Series Data Augmentation for Deep Learning: A Survey](https://mlsanity.com/paper/2002.12478/) (2020)
- [Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks](https://mlsanity.com/paper/1806.05393/) (2018)
- [Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learning](https://mlsanity.com/paper/2110.03909/) (2021)
- [Learning to Teach with Dynamic Loss Functions](https://mlsanity.com/paper/1810.12081/) (2018)
- [Addressing the Loss-Metric Mismatch with Adaptive Loss Alignment](https://mlsanity.com/paper/1905.05895/) (2019)
- [Improved Training Speed, Accuracy, and Data Utilization Through Loss Function Optimization](https://mlsanity.com/paper/1905.11528/) (2019)
- [L2T-DLN: Learning to Teach with Dynamic Loss Network](https://mlsanity.com/paper/2310.19313/) (2023)

## Abstract (arXiv)

We propose Evolutionary Dynamic Loss (EDL), a framework that learns a transferable classification loss in the probability space using unlimited synthetic prediction-label pairs, without accessing real samples during the main loss pretraining stage. EDL parameterizes the loss as a lightweight network and is trained with a semantics-free ranking-consistency objective that assigns larger penalties for more erroneous predictions. To robustly explore the space of loss functions, we optimize EDL via an evolutionary strategy and introduce chaotic mutation to improve exploration under noisy fitness evaluations. Experiments on CIFAR-10 with ResNet backbones show that EDL can serve as a drop-in replacement for cross-entropy and achieves competitive or improved accuracy, while ablation studies confirm that chaotic mutation yields faster convergence and better synthetic pretraining metrics than standard Gaussian mutation.