mlsanityData to 14 Oct 2026

AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks

arXiv 2103.14026 ↗First posted 25 Mar 2021cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

Significant progress has been achieved in automating the design of various components in deep networks. However, the automatic design of loss functions for generic tasks with various evaluation metrics remains under-investigated. Previous works on handcrafting loss functions heavily rely on human expertise, which limits their extendibility. Meanwhile, existing efforts on searching loss functions mainly focus on specific tasks and particular metrics, with task-specific heuristics. Whether such works can be extended to generic tasks is not verified and questionable. In this paper, we propose AutoLoss-Zero, the first general framework for searching loss functions from scratch for generic tasks. Specifically, we design an elementary search space composed only of primitive mathematical operators to accommodate the heterogeneous tasks and evaluation metrics. A variant of the evolutionary algorithm is employed to discover loss functions in the elementary search space. A loss-rejection protocol and a gradient-equivalence-check strategy are developed so as to improve the search efficiency, which are applicable to generic tasks. Extensive experiments on various computer vision tasks demonstrate that our searched loss functions are on par with or superior to existing loss functions, which generalize well to different datasets and networks. Code shall be released.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 46 of its 67 references that are in the map. First posted on arXiv 25 Mar 2021.

Builds on

46 of 67 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2017Mask R-CNNcited by 7,277
  7. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  9. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  10. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  11. 2015Fast R-CNNcited by 2,933
  12. 2016Pyramid Scene Parsing Networkcited by 2,479
34 more
  1. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  3. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  4. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  5. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,368
  6. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  7. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  8. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  9. 2018Simple Baselines for Human Pose Estimation and Trackingcited by 502
  10. 2019Distance-IoU Loss: Faster and Better Learning for Bounding Box Regressioncited by 459
  11. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 429
  12. 2019AutoML: A Survey of the State-of-the-Artcited by 390
  13. 2020A Comprehensive Survey of Neural Architecture Search: Challenges and Solutionscited by 254
  14. 2016UnitBox: An Advanced Object Detection Networkcited by 205
  15. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 159
  16. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 143
  17. 2015Minimum Risk Training for Neural Machine Translationcited by 115
  18. 2018Gradient Harmonized Single-stage Detectorcited by 114
  19. 2020Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentationcited by 98
  20. 2018A Study of Reinforcement Learning for Neural Machine Translationcited by 74
  21. 2020Evolving Normalization-Activation Layerscited by 45
  22. 2020FBNetV3: Joint Architecture-Recipe Search using Predictor Pretrainingcited by 35
  23. 2021AutoDropout: Learning Dropout Patterns to Regularize Deep Networkscited by 35
  24. 2019AM-LFS: AutoML for Loss Function Searchcited by 29
  25. 2019Distance Map Loss Penalty Term for Semantic Segmentationcited by 27
  26. 2015Training Deep Neural Networks via Direct Loss Minimizationcited by 25
  27. 2020A Ranking-based, Balanced Loss Function Unifying Classification and Localisation in Object Detectioncited by 25
  28. 2019DR Loss: Improving Object Detection by Distributional Rankingcited by 24
  29. 2020AP-Loss for Accurate One-Stage Object Detectioncited by 24
  30. 2020Loss Function Search for Face Recognitioncited by 22
  31. 2016Bridging Category-level and Instance-level Semantic Image Segmentationcited by 21
  32. 2021Loss Function Discovery for Object Detection via Convergence-Simulation Driven Searchcited by 20
  33. 2020Auto Seg-Loss: Searching Metric Surrogates for Semantic Segmentationcited by 16
  34. 2016Efficient Optimization for Rank-based Loss Functionscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 3; 2023: 4; 2024: 6; 2025: 1; 2026: 2.

19 papers.

Sort:

By area:

  1. 2023EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantizationcited by 22
  2. 2021Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regressioncited by 18
  3. 2024Automated data processing and feature engineering for deep learning and big data applications: a surveycited by 14
  4. 2022Reinforcement Learning with Automated Auxiliary Loss Searchcited by 13
  5. 2023Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discoverycited by 13
  6. 2023Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labelscited by 12
  7. 2022AutoLossGen: Automatic Loss Function Generation for Recommender Systemscited by 10
  8. 2022Learning Symbolic Model-Agnostic Loss Functions via Meta-Learningcited by 9
  9. 2024LPZero: Language Model Zero-cost Proxy Search from Zerocited by 7
  10. 2023Meta-Learning Adaptive Loss Functionscited by 4
  11. 2021Tightening the Approximation Error of Adversarial Risk with Auto Loss Function Searchcited by 3
  12. 2024Neural Loss Function Evolution for Large-Scale Image Classifier Convolutional Neural Networkscited by 2
7 more
  1. 2024Next Generation Loss Function for Image Classificationcited by 2
  2. 2021Searching Parameterized AP Loss for Object Detectioncited by 1
  3. 2024Fast and Efficient Local Search for Genetic Programming Based Loss Function Learningcited by 1
  4. 2025MOS-Attack: A Scalable Multi-objective Adversarial Attack Frameworkcited by 1
  5. 2024Evolving Loss Functions for Specific Image Augmentation Techniquescited by 0
  6. 2026AutoPref: Automatic Discovery of Task-Specific Preference Objectives for Neural Combinatorial Optimizationcited by 0
  7. 2026EvoRank: LLM-Guided Evolution of Multi-Objective Learning-to-Rank Pipelinescited by 0