# Large Margin Deep Networks for Classification

arXiv: https://arxiv.org/abs/1803.05598 · HTML: https://mlsanity.com/paper/1803.05598/

## Key facts

- Cited by 112 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2018.

## Built on by (most cited first, 30 of 112)

- [FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding](https://mlsanity.com/paper/2103.05950/) (2021)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [Learning with Fantasy: Semantic-Aware Virtual Contrastive Constraint for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2304.00426/) (2023)
- [A Comprehensive Evaluation Framework for Deep Model Robustness](https://mlsanity.com/paper/2101.09617/) (2021)
- [LESS: Label-Efficient Semantic Segmentation for LiDAR Point Clouds](https://mlsanity.com/paper/2210.08064/) (2022)
- [Unbiased Supervised Contrastive Learning](https://mlsanity.com/paper/2211.05568/) (2022)
- [Why neural networks find simple solutions: the many regularizers of geometric complexity](https://mlsanity.com/paper/2209.13083/) (2022)
- [Exploring and Exploiting Decision Boundary Dynamics for Adversarial Robustness](https://mlsanity.com/paper/2302.03015/) (2023)
- [Reverse Engineering Self-Supervised Learning](https://mlsanity.com/paper/2305.15614/) (2023)
- [Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition](https://mlsanity.com/paper/2206.06629/) (2022)
- [Test time Adaptation through Perturbation Robustness](https://mlsanity.com/paper/2110.10232/) (2021)
- [OODformer: Out-Of-Distribution Detection Transformer](https://mlsanity.com/paper/2107.08976/) (2021)
- [Noisy Feature Mixup](https://mlsanity.com/paper/2110.02180/) (2021)
- [Measuring Generalization with Optimal Transport](https://mlsanity.com/paper/2106.03314/) (2021)
- [Optimizing Bi-Encoder for Named Entity Recognition via Contrastive Learning](https://mlsanity.com/paper/2208.14565/) (2022)
- [Semi-TCL: Semi-Supervised Track Contrastive Representation Learning](https://mlsanity.com/paper/2107.02396/) (2021)
- [Dual Contrastive Learning: Text Classification via Label-Aware Data Augmentation](https://mlsanity.com/paper/2201.08702/) (2022)
- [Certified Robustness via Dynamic Margin Maximization and Improved Lipschitz Regularization](https://mlsanity.com/paper/2310.00116/) (2023)
- [Nearest Class-Center Simplification through Intermediate Layers](https://mlsanity.com/paper/2201.08924/) (2022)
- [Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data](https://mlsanity.com/paper/2202.02842/) (2022)
- [Margin Calibration for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2112.07225/) (2021)
- [Learning Structured Representations with Hyperbolic Embeddings](https://mlsanity.com/paper/2412.01023/) (2024)
- [FedImpro: Measuring and Improving Client Update in Federated Learning](https://mlsanity.com/paper/2402.07011/) (2024)
- [Understanding Square Loss in Training Overparametrized Neural Network Classifiers](https://mlsanity.com/paper/2112.03657/) (2021)
- [Task Agnostic Representation Consolidation: a Self-supervised based Continual Learning Approach](https://mlsanity.com/paper/2207.06267/) (2022)
- [Towards Bridging Generalization and Expressivity of Graph Neural Networks](https://mlsanity.com/paper/2410.10051/) (2024)
- [Label Contrastive Coding based Graph Neural Network for Graph Classification](https://mlsanity.com/paper/2101.05486/) (2021)
- [Deep Learning Generalization and the Convex Hull of Training Sets](https://mlsanity.com/paper/2101.09849/) (2021)
- [MarginMatch: Improving Semi-Supervised Learning with Pseudo-Margins](https://mlsanity.com/paper/2308.09037/) (2023)

## Abstract (arXiv)

We present a formulation of deep learning that aims at producing a large margin classifier. The notion of margin, minimum distance to a decision boundary, has served as the foundation of several theoretically profound and empirically successful results for both classification and regression tasks. However, most large margin algorithms are applicable only to shallow models with a preset feature representation; and conventional margin methods for neural networks only enforce margin at the output layer. Such methods are therefore not well suited for deep networks. In this work, we propose a novel loss function to impose a margin on any chosen set of layers of a deep network (including input and hidden layers). Our formulation allows choosing any norm on the metric measuring the margin. We demonstrate that the decision boundary obtained by our loss has nice properties compared to standard classification loss functions. Specifically, we show improved empirical results on the MNIST, CIFAR-10 and ImageNet datasets on multiple tasks: generalization from small training sets, corrupted labels, and robustness against adversarial perturbations. The resulting loss is general and complementary to existing data augmentation (such as random/adversarial input transform) and regularization techniques (such as weight decay, dropout, and batch norm).