mlsanityData to 14 Oct 2026

BayesFormer: Transformer with Uncertainty Estimation

arXiv 2206.00826 ↗First posted 2 Jun 2022cs.CL cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Transformer has become ubiquitous due to its dominant performance in various NLP and image processing tasks. However, it lacks understanding of how to generate mathematically grounded uncertainty estimates for transformer architectures. Models equipped with such uncertainty estimates can typically improve predictive performance, make networks robust, avoid over-fitting and used as acquisition function in active learning. In this paper, we introduce BayesFormer, a Transformer model with dropouts designed by Bayesian theory. We proposed a new theoretical framework to extend the approximate variational inference-based dropout to Transformer-based architectures. Through extensive experiments, we validate the proposed architecture in four paradigms and show improvements across the board: language modeling and classification, long-sequence understanding, machine translation and acquisition function for active learning.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 43 of its 70 references that are in the map. First posted on arXiv 2 Jun 2022.

Builds on

43 of 70 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  11. 2020Reformer: The Efficient Transformercited by 1,668
  12. 2020Rethinking Attention with Performerscited by 1,601
31 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  4. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  6. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 964
  7. 2018Neural Network Acceptability Judgmentscited by 958
  8. 2017Deep Bayesian Active Learning with Image Datacited by 819
  9. 2020Efficient Transformers: A Surveycited by 790
  10. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  11. 2020A Survey on Visual Transformercited by 654
  12. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 607
  13. 2016Deep Exploration via Bootstrapped DQNcited by 527
  14. 2019Introduction to Multi-Armed Banditscited by 481
  15. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
  16. 2017A Tutorial on Thompson Samplingcited by 433
  17. 2021A Survey of Transformerscited by 401
  18. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
  19. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 129
  20. 2017A Deep Relevance Matching Model for Ad-hoc Retrievalcited by 126
  21. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 113
  22. 2022Transformer for Graphs: An Overview from Architecture Perspectivecited by 107
  23. 2018Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Studycited by 90
  24. 2016Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriorscited by 82
  25. 2022Uncertainty Estimation for Language Reward Modelscited by 37
  26. 2019Disentangling neural mechanisms for perceptual groupingcited by 28
  27. 2021Stochastic Bandits for Multi-platform Budget Optimization in Online Advertisingcited by 25
  28. 2021Bayesian Transformer Language Models for Speech Recognitioncited by 22
  29. 2013Model-Based Bayesian Explorationcited by 21
  30. 2020Minimax Active Learningcited by 18
  31. 2021UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Costcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 2; 2025: 7; 2026: 6.

19 papers.

Sort:

By area:

  1. 2023A Categorical Archive of ChatGPT Failurescited by 168
  2. 2023LoRA ensembles for large language model fine-tuningcited by 67
  3. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  4. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 31
  5. 2024EVCL: Elastic Variational Continual Learning with Weight Consolidationcited by 9
  6. 2025Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformerscited by 6
  7. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 4
  8. 2022To Softmax, or not to Softmax: that is the question when applying Active Learning for Transformer Modelscited by 3
  9. 2026Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Modelscited by 2
  10. 2026In Transformer We Trust? A Perspective on Transformer Architecture Failure Modescited by 1
  11. 2026Variational Neurons in Transformers for Language Modelingcited by 1
  12. 2024Evidential Transformers for Improved Image Retrievalcited by 0
7 more
  1. 2025E-BayesSAM: Efficient Bayesian Adaptation of SAM with Self-Optimizing KAN-Based Interpretation for Uncertainty-Aware Ultrasonic Segmentationcited by 0
  2. 2025A Multi-target Bayesian Transformer Framework for Predicting Cardiovascular Disease Biomarkers during Pandemicscited by 0
  3. 2025Kalman Bayesian Transformercited by 0
  4. 2025GRASP: GRouped Activation Shared Parameterization for Parameter-Efficient Fine-Tuning and Robust Inference of Transformerscited by 0
  5. 2026Agentic Control in Variational Language Modelscited by 0
  6. 2026Uncertainty Estimation in Instance Segmentation of Affordances via Bayesian Visual Transformerscited by 0
  7. 2026Sound Probabilistic Safety Bounds for Large Language Modelscited by 0