mlsanityData to 7 Oct 2026

Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models

arXiv 2109.05793 ↗First posted 13 Sep 2021cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Recent works have shown that powerful pre-trained language models (PLM) can be fooled by small perturbations or intentional attacks. To solve this issue, various data augmentation techniques are proposed to improve the robustness of PLMs. However, it is still challenging to augment semantically relevant examples with sufficient diversity. In this work, we present Virtual Data Augmentation (VDA), a general framework for robustly fine-tuning PLMs. Based on the original token embeddings, we construct a multinomial mixture for augmenting virtual data embeddings, where a masked language model guarantees the semantic relevance and the Gaussian noise provides the augmentation diversity. Furthermore, a regularized training strategy is proposed to balance the two aspects. Extensive experiments on six datasets show that our approach is able to improve the robustness of PLMs and alleviate the performance degradation under adversarial attacks. Our codes and data are publicly available at textcolorblueurlhttps://github.com/RUCAIBox/VDA.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 40 of its 57 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

40 of 57 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2016Adversarial examples in the physical worldcited by 1,762
  9. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  10. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  11. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  12. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
28 more
  1. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  2. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  3. 2018Generating Natural Language Adversarial Examplescited by 422
  4. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  5. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  6. 2018Adversarially Robust Generalization Requires More Datacited by 318
  7. 2020S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximizationcited by 311
  8. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  9. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  10. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  11. 2018Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relationscited by 201
  12. 2016Improving the Robustness of Deep Neural Networks via Stability Trainingcited by 156
  13. 2019Adversarial Robustness through Local Linearizationcited by 146
  14. 2017Data Augmentation for Low-Resource Neural Machine Translationcited by 139
  15. 2020Contextualized Perturbation for Textual Adversarial Attackcited by 136
  16. 2018Rademacher Complexity for Adversarially Robust Generalizationcited by 132
  17. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  18. 2020Towards Topic-Guided Conversational Recommender Systemcited by 94
  19. 2020Syntactic Data Augmentation Increases Robustness to Inference Heuristicscited by 85
  20. 2018Sequence-to-Sequence Data Augmentation for Dialogue Language Understandingcited by 67
  21. 2015Distributional Smoothing with Virtual Adversarial Trainingcited by 61
  22. 2020Robust Encodings: A Framework for Combating Adversarial Typoscited by 58
  23. 2019On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Modelscited by 56
  24. 2021Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!cited by 51
  25. 2020Generating Fluent Adversarial Examples for Natural Languagescited by 49
  26. 2018Robust Machine Comprehension Models via Adversarial Trainingcited by 44
  27. 2018Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Datacited by 37
  28. 2019Unsupervised Context Rewriting for Open Domain Conversationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 1; 2025: 1; 2026: 1.

5 papers.

  1. 2022EPiDA: An Easy Plug-in Data Augmentation Framework for High Performance Text Classificationcited by 7
  2. 2023Test-Time Adaptation with Perturbation Consistency Learningcited by 0
  3. 2024Evaluation Metrics for Text Data Augmentation in NLPcited by 0
  4. 2025Intended Target Identification for Anomia Patients with Gradient-based Selective Augmentationcited by 0
  5. 2026ToxiTrace: Gradient-Aligned Training for Explainable Chinese Toxicity Detectioncited by 0