mlsanityData to 7 Oct 2026

Learning Symbolic Model-Agnostic Loss Functions via Meta-Learning

arXiv 2209.08907 ↗First posted 19 Sep 2022cs.LG cs.NEcited by 9

Not on the map. Why?

Abstract (arXiv)

In this paper, we develop upon the emerging topic of loss function learning, which aims to learn loss functions that significantly improve the performance of the models trained under them. Specifically, we propose a new meta-learning framework for learning model-agnostic loss functions via a hybrid neuro-symbolic search approach. The framework first uses evolution-based methods to search the space of primitive mathematical operations to find a set of symbolic loss functions. Second, the set of learned loss functions are subsequently parameterized and optimized via an end-to-end gradient-based training procedure. The versatility of the proposed framework is empirically validated on a diverse set of supervised learning tasks. Results show that the meta-learned loss functions discovered by the newly proposed method outperform both the cross-entropy loss and state-of-the-art loss function learning methods on a diverse range of neural network architectures and datasets.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 52 of its 101 references that are in the map. First posted on arXiv 19 Sep 2022.

Builds on

52 of 101 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2016Wide Residual Networkscited by 3,057
  7. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  10. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  11. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  12. 2019When Does Label Smoothing Help?cited by 985
38 more
  1. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  2. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  3. 2016Learning to learn by gradient descent by gradient descentcited by 616
  4. 2019Meta-Learning with Implicit Gradientscited by 523
  5. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  6. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 403
  7. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 377
  8. 2015Recent Advances in Convolutional Neural Networkscited by 360
  9. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  10. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  11. 2018Meta-Learning: A Surveycited by 254
  12. 2017A General and Adaptive Robust Loss Functioncited by 236
  13. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 232
  14. 2018Truncated Back-propagation for Bilevel Optimizationcited by 187
  15. 2016Deep Pyramidal Residual Networkscited by 163
  16. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 137
  17. 2019Feature-Critic Networks for Heterogeneous Domain Generalizationcited by 130
  18. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 117
  19. 2019Generalized Inner Loop Meta-Learningcited by 92
  20. 2016Low Data Drug Discovery with One-shot Learningcited by 91
  21. 2018Evolved Policy Gradientscited by 86
  22. 2019AI Benchmark: All About Deep Learning on Smartphones in 2019cited by 76
  23. 2019Meta-Learning of Neural Architectures for Few-Shot Learningcited by 73
  24. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 68
  25. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  26. 2022PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functionscited by 62
  27. 2019Meta-Learning via Learned Losscited by 60
  28. 2019ES-MAML: Simple Hessian-Free Meta Learningcited by 57
  29. 2021Meta-learning PINN loss functionscited by 51
  30. 2021Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learningcited by 49
  31. 2021Evolving Reinforcement Learning Algorithmscited by 48
  32. 2018Learning to Teach with Dynamic Loss Functionscited by 40
  33. 2019Addressing the Loss-Metric Mismatch with Adaptive Loss Alignmentcited by 33
  34. 2019Improved Training Speed, Accuracy, and Data Utilization Through Loss Function Optimizationcited by 28
  35. 2019Learning Surrogate Lossescited by 21
  36. 2021Loss Function Discovery for Object Detection via Convergence-Simulation Driven Searchcited by 20
  37. 2021AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Taskscited by 19
  38. 2020A Comprehensive Overview and Survey of Recent Advances in Meta-Learningcited by 15

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 6; 2025: 2.

9 papers.

  1. 2025Neuro-Symbolic AI in 2024: A Systematic Reviewcited by 36
  2. 2023Meta-Learning Adaptive Loss Functionscited by 4
  3. 2024Neural Loss Function Evolution for Large-Scale Image Classifier Convolutional Neural Networkscited by 2
  4. 2024STAA-Net: A Sparse and Transferable Adversarial Attack for Speech Emotion Recognitioncited by 1
  5. 2024Sharpness-Aware Minimization for Evolutionary Feature Construction in Regressioncited by 1
  6. 2024Meta-Learning Neural Procedural Biasescited by 1
  7. 2024A Consistent Lebesgue Measure for Multi-label Learningcited by 0
  8. 2024Evolving Loss Functions for Specific Image Augmentation Techniquescited by 0
  9. 2025Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learningcited by 0