mlsanityData to 7 Oct 2026

Regularizing Transformers With Deep Probabilistic Layers

arXiv 2108.10764 ↗First posted 23 Aug 2021cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Language models (LM) have grown with non-stop in the last decade, from sequence-to-sequence architectures to the state-of-the-art and utter attention-based Transformers. In this work, we demonstrate how the inclusion of deep generative models within BERT can bring more versatile models, able to impute missing/noisy words with richer text or even improve BLEU score. More precisely, we use a Gaussian Mixture Variational Autoencoder (GMVAE) as a regularizer layer and prove its effectiveness not only in Transformers but also in the most relevant encoder-decoder based LM, seq2seq with and without attention.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 54 of its 82 references that are in the map. First posted on arXiv 23 Aug 2021.

Builds on

54 of 82 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2013Auto-Encoding Variational Bayescited by 3,980
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
38 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  4. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  5. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  6. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  7. 2018Self-Attention Generative Adversarial Networkscited by 809
  8. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  9. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  10. 2015Generating Sentences from a Continuous Spacecited by 607
  11. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  12. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  13. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  14. 2019Revealing the Dark Secrets of BERTcited by 300
  15. 2017Toward Controlled Generation of Textcited by 271
  16. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  17. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 230
  18. 2016Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoderscited by 219
  19. 2016Variational Deep Embedding: An Unsupervised and Generative Approach to Clusteringcited by 219
  20. 2019Levenshtein Transformercited by 197
  21. 2021TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Upcited by 162
  22. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 149
  23. 2020Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Spacecited by 114
  24. 2017Long Text Generation via Adversarial Training with Leaked Informationcited by 100
  25. 2018Language GANs Falling Shortcited by 97
  26. 2018MaskGAN: Better Text Generation via Filling in the______cited by 89
  27. 2017Cold Fusion: Training Seq2Seq Models Together with Language Modelscited by 87
  28. 2013Early stopping and non-parametric regression: An optimal data-dependent stopping rulecited by 86
  29. 2019A Tensorized Transformer for Language Modelingcited by 85
  30. 2017Improved Variational Autoencoders for Text Modeling using Dilated Convolutionscited by 81
  31. 2020ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corporacited by 63
  32. 2017Adversarial Feature Matching for Text Generationcited by 58
  33. 2017A Deep Generative Framework for Paraphrase Generationcited by 54
  34. 2019Evaluating Word Embedding Models: Methods and Experimental Resultscited by 37
  35. 2019Training language GANs from Scratchcited by 31
  36. 2018Conditional Variational Autoencoder for Neural Machine Translationcited by 20
  37. 2019TextKD-GAN: Text Generation using KnowledgeDistillation and Generative Adversarial Networkscited by 14
  38. 2018Adversarial Text Generation Without Reinforcement Learningcited by 7

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2024Faraday: Synthetic Smart Meter Generator for the smart gridcited by 6
  2. 2025Self-supervised Latent Space Optimization with Nebula Variational Codingcited by 1
  3. 2025Crisp Attention: Regularizing Transformers via Structured Sparsitycited by 1
  4. 2025AttentionDrop: A Novel Regularization Method for Transformer Modelscited by 0