# Deep Learning using Linear Support Vector Machines

arXiv: https://arxiv.org/abs/1306.0239 · HTML: https://mlsanity.com/paper/1306.0239/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2013.

## Built on by (most cited first, 30 of 39)

- [Facial Expression Recognition with Visual Transformers and Attentional Selective Fusion](https://mlsanity.com/paper/2103.16854/) (2021)
- [Facial Emotion Recognition: State of the Art Performance on FER2013](https://mlsanity.com/paper/2105.03588/) (2021)
- [On the Equivalence between Neural Network and Support Vector Machine](https://mlsanity.com/paper/2111.06063/) (2021)
- [Portrait Eyeglasses and Shadow Removal by Leveraging 3D Synthetic Data](https://mlsanity.com/paper/2203.10474/) (2022)
- [Teacher-Student Training and Triplet Loss to Reduce the Effect of Drastic Face Occlusion](https://mlsanity.com/paper/2111.10561/) (2021)
- [FER-former: Multi-modal Transformer for Facial Expression Recognition](https://mlsanity.com/paper/2303.12997/) (2023)
- [Recent Advances in Large Margin Learning](https://mlsanity.com/paper/2103.13598/) (2021)
- [Face Trees for Expression Recognition](https://mlsanity.com/paper/2112.02487/) (2021)
- [Glo-In-One: Holistic Glomerular Detection, Segmentation, and Lesion Characterization with Large-scale Web Image Mining](https://mlsanity.com/paper/2206.00123/) (2022)
- [New SAR target recognition based on YOLO and very deep multi-canonical correlation analysis](https://mlsanity.com/paper/2110.15383/) (2021)
- [FaceTopoNet: Facial Expression Recognition using Face Topology Learning](https://mlsanity.com/paper/2209.06322/) (2022)
- [SegNet: A Segmented Deep Learning based Convolutional Neural Network Approach for Drones Wildfire Detection](https://mlsanity.com/paper/2405.00031/) (2024)
- [Regional Attention Network (RAN) for Head Pose and Fine-grained Gesture Recognition](https://mlsanity.com/paper/2101.06634/) (2021)
- [Self-Supervised Interpretable End-to-End Learning via Latent Functional Modularity](https://mlsanity.com/paper/2403.18947/) (2024)
- [Quantum Machine Learning with HQC Architectures using non-Classically Simulable Feature Maps](https://mlsanity.com/paper/2103.11381/) (2021)
- [Kernel function impact on convolutional neural networks](https://mlsanity.com/paper/2302.10266/) (2023)
- [A Mirror Descent-Based Algorithm for Corruption-Tolerant Distributed Gradient Descent](https://mlsanity.com/paper/2407.14111/) (2024)
- [Exploring Kernel Transformations for Implicit Neural Representations](https://mlsanity.com/paper/2504.04728/) (2025)
- [MAFER: a Multi-resolution Approach to Facial Expression Recognition](https://mlsanity.com/paper/2105.02481/) (2021)
- [Recognizing Facial Expressions in the Wild using Multi-Architectural Representations based Ensemble Learning with Distillation](https://mlsanity.com/paper/2106.16126/) (2021)
- [Algorithms for the Training of Neural Support Vector Machines](https://mlsanity.com/paper/2308.07204/) (2023)
- [Deep Support Vectors](https://mlsanity.com/paper/2403.17329/) (2024)
- [Large-Margin Hyperdimensional Computing: A Learning-Theoretical Perspective](https://mlsanity.com/paper/2603.03830/) (2026)
- [Classification of COVID-19 X-ray Images Using a Combination of Deep and Handcrafted Features](https://mlsanity.com/paper/2101.07866/) (2021)
- [A Multi-resolution Approach to Expression Recognition in the Wild](https://mlsanity.com/paper/2103.05723/) (2021)
- [A Novel Hybrid Deep Learning Approach for Non-Intrusive Load Monitoring of Residential Appliance Based on Long Short Term Memory and Convolutional Neural Networks](https://mlsanity.com/paper/2104.07809/) (2021)
- [Deep Low-Density Separation for Semi-Supervised Classification](https://mlsanity.com/paper/2205.11995/) (2022)
- [Introducing One Sided Margin Loss for Solving Classification Problems in Deep Networks](https://mlsanity.com/paper/2206.01002/) (2022)
- [Finetuning for Sarcasm Detection with a Pruned Dataset](https://mlsanity.com/paper/2212.12213/) (2022)
- [Neural networks for learning personality traits from natural language](https://mlsanity.com/paper/2302.13782/) (2023)

## Abstract (arXiv)

Recently, fully-connected and convolutional neural networks have been trained to achieve state-of-the-art performance on a wide variety of tasks such as speech recognition, image classification, natural language processing, and bioinformatics. For classification tasks, most of these "deep learning" models employ the softmax activation function for prediction and minimize cross-entropy loss. In this paper, we demonstrate a small but consistent advantage of replacing the softmax layer with a linear support vector machine. Learning minimizes a margin-based loss instead of the cross-entropy loss. While there have been various combinations of neural nets and SVMs in prior art, our results using L2-SVMs show that by simply replacing softmax with linear SVMs gives significant gains on popular deep learning datasets MNIST, CIFAR-10, and the ICML 2013 Representation Learning Workshop's face expression recognition challenge.