mlsanityData to 7 Oct 2026

FreeLB: Enhanced Adversarial Training for Natural Language Understanding

arXiv 1909.11764 ↗First posted 25 Sep 2019cs.CL cs.LGcited by 261

Not on the map. Why?

Abstract (arXiv)

Adversarial training, which minimizes the maximal risk for label-preserving input perturbations, has proved to be effective for improving the generalization of language models. In this work, we propose a novel adversarial training algorithm, FreeLB, that promotes higher invariance in the embedding space, by adding adversarial perturbations to word embeddings and minimizing the resultant adversarial risk inside different regions around input samples. To validate the effectiveness of the proposed approach, we apply it to Transformer-based models for natural language understanding and commonsense reasoning tasks. Experiments on the GLUE benchmark show that when applied only to the finetuning stage, it is able to improve the overall test scores of BERT-base model from 78.3 to 79.4, and RoBERTa-large model from 88.5 to 88.8. In addition, the proposed approach achieves state-of-the-art single-model test accuracies of 85.44% and 67.75% on ARC-Easy and ARC-Challenge. Experiments on CommonsenseQA benchmark further demonstrate that FreeLB can be generalized and boost the performance of RoBERTa-large model on other tasks as well. Code is available at urlhttps://github.com/zhuchen03/FreeLB .

Key facts. Cited by 261 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 63; 2022: 61; 2023: 42; 2024: 57; 2025: 24; 2026: 14.

261 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2021Robust fine-tuning of zero-shot modelscited by 785
  3. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  4. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  5. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  6. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  7. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  8. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  9. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  10. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  11. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  12. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
38 more
  1. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  2. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  3. 2021Advances in adversarial attacks and defenses in computer vision: A surveycited by 127
  4. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  5. 2021A Survey on Data Augmentation for Text Classificationcited by 111
  6. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  7. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  8. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  9. 2021An Empirical Survey of Data Augmentation for Limited Data Learning in NLPcited by 101
  10. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 99
  11. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
  12. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  13. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
  14. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Trainingcited by 88
  15. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  16. 2022One Transformer Can Understand Both 2D & 3D Molecular Datacited by 77
  17. 2021Revisiting and Advancing Fast Adversarial Training Through The Lens of Bi-Level Optimizationcited by 75
  18. 2022Improving Vision Transformers by Revisiting High-frequency Componentscited by 75
  19. 2021Improved OOD Generalization via Adversarial Training and Pre-trainingcited by 66
  20. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60
  21. 2021Certified Robustness to Text Adversarial Attacks by Randomized [MASK]cited by 57
  22. 2021Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processingcited by 57
  23. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  24. 2021Searching for an Effective Defender: Benchmarking Defense against Adversarial Word Substitutioncited by 55
  25. 2021CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understandingcited by 50
  26. 2021Adversarial Attacks On Multi-Agent Communicationcited by 48
  27. 2022Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentationscited by 46
  28. 2022Out-of-distribution Generalization with Causal Invariant Transformationscited by 45
  29. 2022SemAttack: Natural Textual Attacks via Different Semantic Spacescited by 42
  30. 2021Consistency Regularization for Cross-Lingual Fine-Tuningcited by 41
  31. 2021JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answeringcited by 40
  32. 2021SSMix: Saliency-Based Span Mixup for Text Classificationcited by 39
  33. 2021Improved Text Classification via Contrastive Adversarial Trainingcited by 38
  34. 2023NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Modelscited by 33
  35. 2021HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizationcited by 31
  36. 2021Defending Against Backdoor Attacks in Natural Language Generationcited by 30
  37. 2021Noise Stability Regularization for Improving BERT Fine-tuningcited by 29
  38. 2021Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspectivecited by 25

50 most cited of 261 All 261 (JSON)