mlsanityData to 7 Oct 2026

Recent Advances in Large Margin Learning

arXiv 2103.13598 ↗First posted 25 Mar 2021cs.LG cs.CR cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

This paper serves as a survey of recent advances in large margin training and its theoretical foundations, mostly for (nonlinear) deep neural networks (DNNs) that are probably the most prominent machine learning models for large-scale data in the community over the past decade. We generalize the formulation of classification margins from classical research to latest DNNs, summarize theoretical connections between the margin, network generalization, and robustness, and introduce recent efforts in enlarging the margins for DNNs comprehensively. Since the viewpoint of different methods is discrepant, we categorize them into groups for ease of comparison and discussion in the paper. Hopefully, our discussions and overview inspire new research work in the community that aim to improve the performance of DNNs, and we also point to directions where the large margin principle can be verified to provide theoretical evidence why certain regularizations for DNNs function well in practice. We managed to shorten the paper such that the crucial spirit of large margin learning and related methods are better emphasized.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 82 of its 118 references that are in the map. First posted on arXiv 25 Mar 2021.

Builds on

82 of 118 references.

References · 82

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  9. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  10. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  11. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  12. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
38 more
  1. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  2. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  3. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  4. 2018CosFace: Large Margin Cosine Loss for Deep Face Recognitioncited by 968
  5. 2016Adversarial Machine Learning at Scalecited by 934
  6. 2016Universal adversarial perturbationscited by 774
  7. 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 739
  8. 2017SphereFace: Deep Hypersphere Embedding for Face Recognitioncited by 733
  9. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  10. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  11. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  12. 2017Exploring Generalization in Deep Learningcited by 595
  13. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 577
  14. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  15. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 564
  16. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  17. 2018Efficient Neural Network Robustness Certification with General Activation Functionscited by 439
  18. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  19. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  20. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 301
  21. 2016Large-Margin Softmax Loss for Convolutional Neural Networkscited by 296
  22. 2017Adversarial Examples for Semantic Segmentation and Object Detectioncited by 275
  23. 2018Additive Margin Softmax for Face Verificationcited by 275
  24. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  25. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
  26. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  27. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 211
  28. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 193
  29. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  30. 2017Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulationcited by 182
  31. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  32. 2018Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approachcited by 181
  33. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  34. 2018Robustness via curvature regularization, and vice versacited by 174
  35. 2018MMA Training: Direct Input Space Margin Maximization through Adversarial Trainingcited by 171
  36. 2017Generalization in Deep Learningcited by 146
  37. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  38. 2018A Dual Approach to Scalable Verification of Deep Networkscited by 125

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 4.

7 papers.

  1. 2024A Systematic Review on Long-Tailed Learningcited by 27
  2. 2023Certified Robustness via Dynamic Margin Maximization and Improved Lipschitz Regularizationcited by 19
  3. 2023Certified Robust Models with Slack Control and Large Lipschitz Constantscited by 3
  4. 2024Edge Detectors Can Make Deep Convolutional Neural Networks More Robustcited by 2
  5. 2024Marginal Debiased Network for Fair Visual Recognitioncited by 1
  6. 2023Dendritic Integration Based Quadratic Neural Networks Outperform Traditional Aritificial Onescited by 0
  7. 2024Potential Energy based Mixture Model for Noisy Label Learningcited by 0