mlsanityData to 7 Oct 2026

Consistency Training with Virtual Adversarial Discrete Perturbation

arXiv 2104.07284 ↗First posted 15 Apr 2021cs.CL cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Consistency training regularizes a model by enforcing predictions of original and perturbed inputs to be similar. Previous studies have proposed various augmentation methods for the perturbation but are limited in that they are agnostic to the training model. Thus, the perturbed samples may not aid in regularization due to their ease of classification from the model. In this context, we propose an augmentation method of adding a discrete noise that would incur the highest divergence between predictions. This virtual adversarial discrete noise obtained by replacing a small portion of tokens while keeping original semantics as much as possible efficiently pushes a training model's decision boundary. Experimental results show that our proposed method outperforms other consistency training baselines with text editing, paraphrasing, or a continuous noise on semi-supervised text classification tasks and a robustness benchmark

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 47 of its 73 references that are in the map. First posted on arXiv 15 Apr 2021.

Builds on

47 of 73 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  12. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
35 more
  1. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  2. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  3. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  4. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  5. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  6. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  7. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  8. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  9. 2016Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learningcited by 441
  10. 2018Generating Natural Language Adversarial Examplescited by 422
  11. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  12. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  13. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  14. 2019Interpolation Consistency Training for Semi-Supervised Learningcited by 248
  15. 2018Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relationscited by 201
  16. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  17. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
  18. 2019Not Enough Data? Deep Learning to the Rescue!cited by 174
  19. 2020Contextualized Perturbation for Textual Adversarial Attackcited by 136
  20. 2019Augmenting Data with Mixup for Sentence Classification: An Empirical Studycited by 125
  21. 2020Adversarial Training for Large Neural Language Modelscited by 109
  22. 2018Conditional BERT Contextual Augmentationcited by 107
  23. 2019Robust Neural Machine Translation with Doubly Adversarial Inputscited by 105
  24. 2020SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustnesscited by 94
  25. 2017Data Noising as Smoothing in Neural Network Language Modelscited by 78
  26. 2020Logic-Guided Data Augmentation and Regularization for Consistent Question Answeringcited by 77
  27. 2016Edinburgh Neural Machine Translation Systems for WMT 16cited by 76
  28. 2018Semi-Supervised Sequence Modeling with Cross-View Trainingcited by 75
  29. 2019A Logic-Driven Framework for Consistency of Neural Modelscited by 64
  30. 2019On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Modelscited by 56
  31. 2019Improving Neural Language Modeling via Adversarial Trainingcited by 49
  32. 2020Generative Data Augmentation for Commonsense Reasoningcited by 49
  33. 2018Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledgecited by 45
  34. 2020Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translationcited by 13
  35. 2020Adversarial Subword Regularization for Robust Neural Machine Translationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 2; 2023: 1; 2024: 3; 2025: 0; 2026: 1.

9 papers.

  1. 2021SSMix: Saliency-Based Span Mixup for Text Classificationcited by 39
  2. 2021Learn to Resolve Conversational Dependency: A Consistency Training Framework for Conversational Question Answeringcited by 27
  3. 2022On the Domain Adaptation and Generalization of Pretrained Language Models: A Surveycited by 22
  4. 2022Improving the Sample Efficiency of Prompt Tuning with Domain Adaptationcited by 15
  5. 2024Importance-Aware Data Augmentation for Document-Level Neural Machine Translationcited by 4
  6. 2023RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Trainingcited by 3
  7. 2024Enhancing Effectiveness and Robustness in a Low-Resource Regime via Decision-Boundary-aware Data Augmentationcited by 1
  8. 2024Generalization Measures for Zero-Shot Cross-Lingual Transfercited by 0
  9. 2026Learning Perturbations to Extrapolate Your LLMcited by 0