# Cross-Entropy Loss and Low-Rank Features Have Responsibility for Adversarial Examples

arXiv: https://arxiv.org/abs/1901.08360 · HTML: https://mlsanity.com/paper/1901.08360/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jan 2019.

## Built on by (most cited first, 17 of 17)

- [Non-IID data and Continual Learning processes in Federated Learning: A long road ahead](https://mlsanity.com/paper/2111.13394/) (2021)
- [Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networks](https://mlsanity.com/paper/2104.05097/) (2021)
- [Logic of Differentiable Logics: Towards a Uniform Semantics of DL](https://mlsanity.com/paper/2303.10650/) (2023)
- [Understanding Square Loss in Training Overparametrized Neural Network Classifiers](https://mlsanity.com/paper/2112.03657/) (2021)
- [Adversarial Attack Framework on Graph Embedding Models with Limited Knowledge](https://mlsanity.com/paper/2105.12419/) (2021)
- [Mere Contrastive Learning for Cross-Domain Sentiment Analysis](https://mlsanity.com/paper/2208.08678/) (2022)
- [KNN-BERT: Fine-Tuning Pre-Trained Models with KNN Classifier](https://mlsanity.com/paper/2110.02523/) (2021)
- [Robustness against Adversarial Attacks in Neural Networks using Incremental Dissipativity](https://mlsanity.com/paper/2111.12906/) (2021)
- [Provable Benefit of Mixup for Finding Optimal Decision Boundaries](https://mlsanity.com/paper/2306.00267/) (2023)
- [LaCViT: A Label-aware Contrastive Fine-tuning Framework for Vision Transformers](https://mlsanity.com/paper/2303.18013/) (2023)
- [UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning](https://mlsanity.com/paper/2103.10773/) (2021)
- [CLCE: An Approach to Refining Cross-Entropy and Contrastive Learning for Optimized Learning Fusion](https://mlsanity.com/paper/2402.14551/) (2024)
- [Estimation and Applications of Quantiles in Deep Binary Classification](https://mlsanity.com/paper/2102.06575/) (2021)
- [When hard negative sampling meets supervised contrastive learning](https://mlsanity.com/paper/2308.14893/) (2023)
- [Stationary Point Losses for Robust Model](https://mlsanity.com/paper/2302.09575/) (2023)
- [Optimal Control Theoretic Neural Optimizer: From Backpropagation to Dynamic Programming](https://mlsanity.com/paper/2510.14168/) (2025)
- [Low Rank Adaptation for Adversarial Perturbation](https://mlsanity.com/paper/2604.27487/) (2026)

## Abstract (arXiv)

State-of-the-art neural networks are vulnerable to adversarial examples; they can easily misclassify inputs that are imperceptibly different than their training and test data. In this work, we establish that the use of cross-entropy loss function and the low-rank features of the training data have responsibility for the existence of these inputs. Based on this observation, we suggest that addressing adversarial examples requires rethinking the use of cross-entropy loss function and looking for an alternative that is more suited for minimization with low-rank features. In this direction, we present a training scheme called differential training, which uses a loss function defined on the differences between the features of points from opposite classes. We show that differential training can ensure a large margin between the decision boundary of the neural network and the points in the training dataset. This larger margin increases the amount of perturbation needed to flip the prediction of the classifier and makes it harder to find an adversarial example with small perturbations. We test differential training on a binary classification task with CIFAR-10 dataset and demonstrate that it radically reduces the ratio of images for which an adversarial example could be found -- not only in the training dataset, but in the test dataset as well.