mlsanityData to 14 Oct 2026

Bayesian Attention Belief Networks

arXiv 2106.05251 ↗First posted 9 Jun 2021cs.LG cs.CL stat.MLcited by 25

Not on the map. Why?

Abstract (arXiv)

Attention-based neural networks have achieved state-of-the-art results on a wide range of tasks. Most such models use deterministic attention while stochastic attention is less explored due to the optimization difficulties or complicated model design. This paper introduces Bayesian attention belief networks, which construct a decoder network by modeling unnormalized attention weights with a hierarchy of gamma distributions, and an encoder network by stacking Weibull distributions with a deterministic-upward-stochastic-downward structure to approximate the posterior. The resulting auto-encoding networks can be optimized in a differentiable way with a variational lower bound. It is simple to convert any models with deterministic attention, including pretrained ones, to the proposed Bayesian attention belief networks. On a variety of language understanding tasks, we show that our method outperforms deterministic attention and state-of-the-art stochastic attention in accuracy, uncertainty estimation, generalization across domains, and robustness to adversarial attacks. We further demonstrate the general applicability of our method on neural machine translation and visual question answering, showing great potential of incorporating our method into various attention-related tasks.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 57 of its 85 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

57 of 85 references.

References · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  8. 2016Layer Normalizationcited by 5,669
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  11. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
38 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  2. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  12. 2016Variational Inference: A Review for Statisticianscited by 1,472
  13. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  14. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  15. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  16. 2018Neural Network Acceptability Judgmentscited by 958
  17. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  18. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  19. 2015Generating Sentences from a Continuous Spacecited by 612
  20. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 381
  21. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  22. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  23. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  24. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  25. 2016Ladder Variational Autoencoderscited by 318
  26. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  27. 2020Calibration of Pre-trained Transformerscited by 315
  28. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 228
  29. 2016A Decomposable Attention Model for Natural Language Inferencecited by 215
  30. 2016Enhanced LSTM for Natural Language Inferencecited by 200
  31. 2018Evaluating Bayesian Deep Learning Methods for Semantic Segmentationcited by 143
  32. 2016Sequential Neural Models with Stochastic Layerscited by 115
  33. 2017Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challengecited by 76
  34. 2017A Continuously Growing Dataset of Sentential Paraphrasescited by 72
  35. 2014Learning Stochastic Recurrent Networkscited by 60
  36. 2018Bayesian Layers: A Module for Neural Network Uncertaintycited by 57
  37. 2017Classical Structured Prediction Losses for Sequence to Sequence Learningcited by 55
  38. 2020Bayesian Attention Modulescited by 51

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 5; 2024: 8; 2025: 3; 2026: 3.

25 papers.

Sort:

By area:

  1. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 115
  2. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 50
  3. 2022Visual Attention Methods in Deep Learning: An In-Depth Surveycited by 37
  4. 2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 37
  5. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  6. 2023POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained modelscited by 29
  7. 2023Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Modelscited by 29
  8. 2022ALLSH: Active Learning Guided by Local Sensitivity and Hardnesscited by 23
  9. 2021Alignment Attention by Matching Key and Query Distributionscited by 14
  10. 2022A Unified Framework for Alternating Offline Model Training and Policy Learningcited by 14
  11. 2024Training-Free Bayesianization for Low-Rank Adapters of Large Language Modelscited by 12
  12. 2021TopicNet: Semantic Graph-Guided Topic Discoverycited by 9
13 more
  1. 2025C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Modelscited by 9
  2. 2022Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Modelscited by 6
  3. 2024Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptationcited by 6
  4. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 4
  5. 2026UAT-LITE: Inference-Time Uncertainty-Aware Attention for Pretrained Transformerscited by 4
  6. 2025Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Modelscited by 3
  7. 2024Enhancing Weakly Supervised 3D Medical Image Segmentation through Probabilistic-aware Learningcited by 2
  8. 2024Towards Robust Uncertainty-Aware Incomplete Multi-View Classificationcited by 1
  9. 2023BayesSpeech: A Bayesian Transformer Network for Automatic Speech Recognitioncited by 0
  10. 2024Switchable Decision: Dynamic Neural Generation Networkscited by 0
  11. 2024Bayesian Low-Rank LeArning (Bella): A Practical Approach to Bayesian Neural Networkscited by 0
  12. 2026Soft Specialists: α-R'enyi Ensembles for Uncertainty-Aware LLM Post-Trainingcited by 0
  13. 2026Scalable Amortized Variational Inference for Non-Poisson Buy-'Til-You-Die Modelscited by 0