mlsanityData to 7 Oct 2026

Rethinking Perturbations in Encoder-Decoders for Fast Training

arXiv 2104.01853 ↗First posted 5 Apr 2021cs.CL cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

We often use perturbations to regularize neural models. For neural encoder-decoders, previous studies applied the scheduled sampling (Bengio et al., 2015) and adversarial perturbations (Sato et al., 2019) as perturbations but these methods require considerable computational time. Thus, this study addresses the question of whether these approaches are efficient enough for training time. We compare several perturbations in sequence-to-sequence problems with respect to computational time. Experimental results show that the simple techniques such as word dropout (Gal and Ghahramani, 2016) and random replacement of input tokens achieve comparable (or better) scores to the recently proposed perturbations, even though these simple methods are faster. Our code is publicly available at https://github.com/takase/rethink_perturbations.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 37 of its 47 references that are in the map. First posted on arXiv 5 Apr 2021.

Builds on

37 of 47 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  9. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  10. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  11. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  12. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
25 more
  1. 2019Green AIcited by 620
  2. 2019Defending Against Neural Fake Newscited by 553
  3. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  4. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  5. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  6. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 221
  7. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  8. 2018Scaling Neural Machine Translationcited by 204
  9. 2018Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relationscited by 201
  10. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
  11. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 147
  12. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  13. 2019Robust Neural Machine Translation with Doubly Adversarial Inputscited by 105
  14. 2017Data Noising as Smoothing in Neural Network Language Modelscited by 78
  15. 2019Tagged Back-Translationcited by 76
  16. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
  17. 2015Distributional Smoothing with Virtual Adversarial Trainingcited by 61
  18. 2019Revisiting Low-Resource Neural Machine Translation: A Case Studycited by 61
  19. 2019Positional Encoding to Control Output Sequence Lengthcited by 56
  20. 2019Improving Neural Language Modeling via Adversarial Trainingcited by 49
  21. 2019Parallel Scheduled Samplingcited by 17
  22. 2017A Nested Attention Neural Hybrid Model for Grammatical Error Correctioncited by 12
  23. 2018Direct Output Connection for a High-Rank Language Modelcited by 10
  24. 2020All Word Embeddings from One Embeddingcited by 8
  25. 2016Cutting-off Redundant Repeating Generations for Neural Abstractive Summarizationcited by 7

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 5; 2023: 4; 2024: 5; 2025: 1.

17 papers.

Sort:

  1. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  2. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  3. 2023ResiDual: Transformer with Dual Residual Connectionscited by 30
  4. 2022B2T Connection: Serving Stability and Performance in Deep Transformerscited by 16
  5. 2022CipherDAug: Ciphertext based Data Augmentation for Neural Machine Translationcited by 9
  6. 2022ExtraPhrase: Efficient Data Augmentation for Abstractive Summarizationcited by 8
  7. 2022Bi-SimCut: A Simple Strategy for Boosting Neural Machine Translationcited by 8
  8. 2024Importance-Aware Data Augmentation for Document-Level Neural Machine Translationcited by 4
  9. 2024Head-wise Shareable Attention for Large Language Modelscited by 4
  10. 2022Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translationcited by 2
  11. 2023PAMI: partition input and aggregate outputs for model interpretationcited by 2
  12. 2024Enhancing Transformer RNNs with Multiple Temporal Perspectivescited by 2
5 more
  1. 2021Doubly-Trained Adversarial Data Augmentation for Neural Machine Translationcited by 1
  2. 2023RenewNAT: Renewing Potential Translation for Non-Autoregressive Transformercited by 1
  3. 2025Self-supervised Latent Space Optimization with Nebula Variational Codingcited by 1
  4. 2024Enhancing Context Through Contrastcited by 0
  5. 2024A Two Dimensional Feature Engineering Method for Relation Extractioncited by 0