mlsanityData to 7 Oct 2026

HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization

arXiv 2106.00149 ↗First posted 31 May 2021cs.CLcited by 31

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large pre-trained models with task-specific data has achieved great success in NLP. However, it has been demonstrated that the majority of information within the self-attention networks is redundant and not utilized effectively during the fine-tuning stage. This leads to inferior results when generalizing the obtained models to out-of-domain distributions. To this end, we propose a simple yet effective data augmentation technique, HiddenCut, to better regularize the model and encourage it to learn more generalizable features. Specifically, contiguous spans within the hidden space are dynamically and strategically dropped during training. Experiments show that our HiddenCut method outperforms the state-of-the-art augmentation methods on the GLUE benchmark, and consistently exhibits superior generalization performances on out-of-distribution and challenging counterexamples. We have publicly released our code at https://github.com/GT-SALT/HiddenCut.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 47 of its 55 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

47 of 55 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2013Intriguing properties of neural networkscited by 5,114
  9. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
35 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  2. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  3. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  4. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  5. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  6. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  7. 2019Adversarial Training for Free!cited by 638
  8. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  9. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  10. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  11. 2019Counterfactual Visual Explanationscited by 312
  12. 2019On the (In)fidelity and Sensitivity for Explanationscited by 309
  13. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  14. 2019Revealing the Dark Secrets of BERTcited by 300
  15. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  16. 2018DropBlock: A regularization method for convolutional networkscited by 261
  17. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  18. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  19. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  20. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  21. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
  22. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  23. 2019Good-Enough Compositional Data Augmentationcited by 158
  24. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  25. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 133
  26. 2016Data Recombination for Neural Semantic Parsingcited by 111
  27. 2020Adversarial Training for Large Neural Language Modelscited by 109
  28. 2020Syntactic Data Augmentation Increases Robustness to Inference Heuristicscited by 85
  29. 2018Semi-Supervised Sequence Modeling with Cross-View Trainingcited by 75
  30. 2020A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generationcited by 72
  31. 2018Robust Machine Comprehension Models via Adversarial Trainingcited by 44
  32. 2021AutoDropout: Learning Dropout Patterns to Regularize Deep Networkscited by 35
  33. 2020Local Additivity Based Data Augmentation for Semi-supervised NERcited by 31
  34. 2018Parameter-free Sentence Embedding via Orthogonal Basiscited by 11
  35. 2020DropCluster: A structured dropout for convolutional networkscited by 7

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 10; 2023: 9; 2024: 4; 2025: 4.

31 papers.

Sort:

  1. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  2. 2021An Empirical Survey of Data Augmentation for Limited Data Learning in NLPcited by 101
  3. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  4. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  5. 2021Data Augmentation Approaches in Natural Language Processing: A Surveycited by 82
  6. 2021Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Modelscited by 76
  7. 2024Prompt Perturbation Consistency Learning for Robust Language Modelscited by 26
  8. 2022Robustness Analysis of Video-Language Models Against Visual and Language Perturbationscited by 23
  9. 2022UnifieR: A Unified Retriever for Large-Scale Retrievalcited by 19
  10. 2024LoRA Meets Dropout under a Unified Frameworkcited by 17
  11. 2023Identifying Semantic Component for Robust Molecular Property Predictioncited by 10
  12. 2022Adversarial Self-Attention for Language Understandingcited by 8
19 more
  1. 2022SDA: Simple Discrete Augmentation for Contrastive Sentence Representation Learningcited by 8
  2. 2022When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentationcited by 5
  3. 2022Data Augmentation for Biomedical Factoid Question Answeringcited by 5
  4. 2022AD-DROP: Attribution-Driven Dropout for Robust Language Model Fine-Tuningcited by 4
  5. 2022ConsistTL: Modeling Consistency in Transfer Learning for Low-Resource Neural Machine Translationcited by 4
  6. 2023RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Trainingcited by 3
  7. 2024Unifying Invariance and Spuriousity for Graph Out-of-Distribution via Probability of Necessity and Sufficiencycited by 3
  8. 2025Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understandingcited by 3
  9. 2022QA Domain Adaptation using Hidden Space Augmentation and Self-Supervised Contrastive Adaptationcited by 2
  10. 2023MixEdit: Revisiting Data Augmentation and Beyond for Grammatical Error Correctioncited by 2
  11. 2024Unifying Invariant and Variant Features for Graph Out-of-Distribution via Probability of Necessity and Sufficiencycited by 2
  12. 2023Adversarial Word Dilution as Text Data Augmentation in Low-Resource Regimecited by 1
  13. 2023Vision-Enhanced Semantic Entity Recognition in Document Images via Visually-Asymmetric Consistency Learningcited by 1
  14. 2023Use the Detection Transformer as a Data Augmentercited by 0
  15. 2023Test-Time Adaptation with Perturbation Consistency Learningcited by 0
  16. 2023Improving the Reusability of Pre-trained Language Models in Real-world Applicationscited by 0
  17. 2025Unified Enhancement of the Generalization and Robustness of Language Models via Bi-Stage Optimizationcited by 0
  18. 2025Intended Target Identification for Anomia Patients with Gradient-based Selective Augmentationcited by 0
  19. 2025Expert-guided Clinical Text Augmentation via Query-Based Model Collaborationcited by 0