# Distributional Smoothing with Virtual Adversarial Training

arXiv: https://arxiv.org/abs/1507.00677 · HTML: https://mlsanity.com/paper/1507.00677/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2015.

## Built on by (most cited first, 30 of 61)

- [Robust Reinforcement Learning on State Observations with Learned Optimal Adversary](https://mlsanity.com/paper/2101.08452/) (2021)
- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [Relating Adversarially Robust Generalization to Flat Minima](https://mlsanity.com/paper/2104.04448/) (2021)
- [Semi-Supervised and Unsupervised Deep Visual Learning: A Survey](https://mlsanity.com/paper/2208.11296/) (2022)
- [Adversarially Guided Actor-Critic](https://mlsanity.com/paper/2102.04376/) (2021)
- [How Smooth Is Attention?](https://mlsanity.com/paper/2312.14820/) (2023)
- [OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers](https://mlsanity.com/paper/2105.14148/) (2021)
- [One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models](https://mlsanity.com/paper/2406.05491/) (2024)
- [Diagnosing Model Performance Under Distribution Shift](https://mlsanity.com/paper/2303.02011/) (2023)
- [Improved Text Classification via Contrastive Adversarial Training](https://mlsanity.com/paper/2107.10137/) (2021)
- [Variational Positive-incentive Noise: How Noise Benefits Models](https://mlsanity.com/paper/2306.07651/) (2023)
- [A Unified Wasserstein Distributional Robustness Framework for Adversarial Training](https://mlsanity.com/paper/2202.13437/) (2022)
- [Can audio-visual integration strengthen robustness under multimodal attacks?](https://mlsanity.com/paper/2104.02000/) (2021)
- [Semi-supervised learning made simple with self-supervised clustering](https://mlsanity.com/paper/2306.07483/) (2023)
- [Salient Objects in Clutter](https://mlsanity.com/paper/2105.03053/) (2021)
- [Rethinking Perturbations in Encoder-Decoders for Fast Training](https://mlsanity.com/paper/2104.01853/) (2021)
- [A soft nearest-neighbor framework for continual semi-supervised learning](https://mlsanity.com/paper/2212.05102/) (2022)
- [Adversarial robustness of amortized Bayesian inference](https://mlsanity.com/paper/2305.14984/) (2023)
- [Towards Adversarially Robust Dataset Distillation by Curvature Regularization](https://mlsanity.com/paper/2403.10045/) (2024)
- [Multi-Expert Adversarial Attack Detection in Person Re-identification Using Context Inconsistency](https://mlsanity.com/paper/2108.09891/) (2021)
- [Global-Local Regularization Via Distributional Robustness](https://mlsanity.com/paper/2203.00553/) (2022)
- [SSLayout360: Semi-Supervised Indoor Layout Estimation from 360-Degree Panorama](https://mlsanity.com/paper/2103.13696/) (2021)
- [Understanding and Improvement of Adversarial Training for Network Embedding from an Optimization Perspective](https://mlsanity.com/paper/2105.08007/) (2021)
- [Random and Adversarial Bit Error Robustness: Energy-Efficient and Secure DNN Accelerators](https://mlsanity.com/paper/2104.08323/) (2021)
- [Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models](https://mlsanity.com/paper/2109.05793/) (2021)
- [Optimal Transport Model Distributional Robustness](https://mlsanity.com/paper/2306.04178/) (2023)
- [Evaluating Model Performance Under Worst-case Subpopulations](https://mlsanity.com/paper/2407.01316/) (2024)
- [Toward Robust Neural Reconstruction from Sparse Point Sets](https://mlsanity.com/paper/2412.16361/) (2024)
- [Multiplicative Reweighting for Robust Neural Network Optimization](https://mlsanity.com/paper/2102.12192/) (2021)
- [Simple Post-Training Robustness Using Test Time Augmentations and Random Forest](https://mlsanity.com/paper/2109.08191/) (2021)

## Abstract (arXiv)

We propose local distributional smoothness (LDS), a new notion of smoothness for statistical model that can be used as a regularization term to promote the smoothness of the model distribution. We named the LDS based regularization as virtual adversarial training (VAT). The LDS of a model at an input datapoint is defined as the KL-divergence based robustness of the model distribution against local perturbation around the datapoint. VAT resembles adversarial training, but distinguishes itself in that it determines the adversarial direction from the model distribution alone without using the label information, making it applicable to semi-supervised learning. The computational cost for VAT is relatively low. For neural network, the approximated gradient of the LDS can be computed with no more than three pairs of forward and back propagations. When we applied our technique to supervised and semi-supervised learning for the MNIST dataset, it outperformed all the training methods other than the current state of the art method, which is based on a highly advanced generative model. We also applied our method to SVHN and NORB, and confirmed our method's superior performance over the current state of the art semi-supervised method applied to these datasets.