mlsanityData to 7 Oct 2026

Mixing between the Cross Entropy and the Expectation Loss Terms

arXiv 2109.05635 ↗First posted 12 Sep 2021cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The cross entropy loss is widely used due to its effectiveness and solid theoretical grounding. However, as training progresses, the loss tends to focus on hard to classify samples, which may prevent the network from obtaining gains in performance. While most work in the field suggest ways to classify hard negatives, we suggest to strategically leave hard negatives behind, in order to focus on misclassified samples with higher probabilities. We show that adding to the optimization goal the expectation loss, which is a better approximation of the zero-one loss, helps the network to achieve better accuracy. We, therefore, propose to shift between the two losses during training, focusing more on the expectation loss gradually during the later stages of training. Our experiments show that the new training protocol improves performance across a diverse set of classification domains, including computer vision, natural language processing, tabular data, and sequences. Our code and scripts are available at supplementary.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 50 references that are in the map. First posted on arXiv 12 Sep 2021.

Builds on

31 of 50 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  8. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  9. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  10. 2016Adversarial examples in the physical worldcited by 1,762
  11. 2017Random Erasing Data Augmentationcited by 1,235
  12. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
19 more
  1. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  2. 2016Adversarial Machine Learning at Scalecited by 934
  3. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  4. 2017Self-Normalizing Neural Networkscited by 591
  5. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 441
  6. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  7. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  8. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  9. 2017Learning From Noisy Large-Scale Datasets With Minimal Supervisioncited by 115
  10. 2017A Hitting Time Analysis of Stochastic Gradient Langevin Dynamicscited by 70
  11. 2017On Loss Functions for Deep Neural Networks in Classificationcited by 68
  12. 2018Imbalanced Deep Learning by Minority Class Incremental Rectificationcited by 65
  13. 2017Variants of RMSProp and Adagrad with Logarithmic Regret Boundscited by 50
  14. 2015Loss Functions for Top-k Error: Analysis and Insightscited by 38
  15. 2017Extending Defensive Distillationcited by 23
  16. 2014Web-Scale Training for Face Identificationcited by 20
  17. 2019Complement Objective Trainingcited by 15
  18. 2018Taming the Cross Entropy Losscited by 10
  19. 2018Statistical Analysis on E-Commerce Reviews, with Sentiment Classification using Bidirectional Recurrent Neural Network (RNN)cited by 8

Built on by

Not cited yet.