mlsanityData to 14 Oct 2026

UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost

arXiv 2104.04946 ↗First posted 11 Apr 2021cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Transformer architecture achieves great success in abundant natural language processing tasks. The over-parameterization of the Transformer model has motivated plenty of works to alleviate its overfitting for superior performances. With some explorations, we find simple techniques such as dropout, can greatly boost model performance with a careful design. Therefore, in this paper, we integrate different dropout techniques into the training of Transformer models. Specifically, we propose an approach named UniDrop to unites three different dropout techniques from fine-grain to coarse-grain, i.e., feature dropout, structure dropout, and data dropout. Theoretically, we demonstrate that these three dropouts play different roles from regularization perspectives. Empirically, we conduct experiments on both neural machine translation and text classification benchmark datasets. Extensive results indicate that Transformer with UniDrop can achieve around 1.5 BLEU improvement on IWSLT14 translation tasks, and better accuracy for the classification even using strong pre-trained RoBERTa as backbone.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 40 of its 55 references that are in the map. First posted on arXiv 11 Apr 2021.

Builds on

40 of 55 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2016Layer Normalizationcited by 5,669
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  12. 2020Reformer: The Efficient Transformercited by 1,668
28 more
  1. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  2. 2016Deep Networks with Stochastic Depthcited by 979
  3. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  4. 2019Are Sixteen Heads Really Better than One?cited by 816
  5. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  6. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  7. 2018Universal Transformerscited by 647
  8. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 464
  9. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  10. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  11. 2018DropBlock: A regularization method for convolutional networkscited by 262
  12. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  13. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  14. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
  15. 2019Robust Learning with Jacobian Regularizationcited by 129
  16. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 113
  17. 2018Improving DNN Robustness to Adversarial Attacks using Jacobian Regularizationcited by 88
  18. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  19. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  20. 2016Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activationscited by 54
  21. 2019Soft Contextual Data Augmentation for Neural Machine Translationcited by 49
  22. 2019Improving Neural Language Modeling via Adversarial Trainingcited by 49
  23. 2020Scheduled DropHead: A Regularization Method for Transformer Modelscited by 28
  24. 2020Guided Transformer: Leveraging Multiple External Sources for Representation Learning in Conversational Searchcited by 23
  25. 2017Weighted Transformer Network for Machine Translationcited by 16
  26. 2020Multi-branch Attentive Transformercited by 10
  27. 2020Alleviating the Inequality of Attention Heads for Neural Machine Translationcited by 6
  28. 2021IOT: Instance-wise Layer Reordering for Transformer Structurescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 3; 2024: 0; 2025: 1; 2026: 1.

11 papers.

  1. 2021Long-Range Transformers for Dynamic Spatiotemporal Forecastingcited by 46
  2. 2021BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translationcited by 35
  3. 2022BayesFormer: Transformer with Uncertainty Estimationcited by 19
  4. 2022CipherDAug: Ciphertext based Data Augmentation for Neural Machine Translationcited by 9
  5. 2022Bi-SimCut: A Simple Strategy for Boosting Neural Machine Translationcited by 8
  6. 2023PASTS: Progress-Aware Spatio-Temporal Transformer Speaker For Vision-and-Language Navigationcited by 8
  7. 2022Relaxed Attention for Transformer Modelscited by 5
  8. 2025Reasoning Bias of Next Token Prediction Trainingcited by 3
  9. 2023EM-Network: Oracle Guided Self-distillation for Sequence Learningcited by 1
  10. 2023Integrating Pre-trained Language Model into Neural Machine Translationcited by 0
  11. 2026Length Generalization Needs Proper Regularizationcited by 0