mlsanityData to 7 Oct 2026

Bayesian Attention Modules

arXiv 2010.10604 ↗First posted 20 Oct 2020stat.ML cs.LG cs.NEcited by 51

Not on the map. Why?

Abstract (arXiv)

Attention modules, as simple and effective tools, have not only enabled deep neural networks to achieve state-of-the-art results in many domains, but also enhanced their interpretability. Most current models use deterministic attention modules due to their simplicity and ease of optimization. Stochastic counterparts, on the other hand, are less popular despite their potential benefits. The main reason is that stochastic attention often introduces optimization issues or requires significant model changes. In this paper, we propose a scalable stochastic version of attention that is easy to implement and optimize. We construct simplex-constrained attention distributions by normalizing reparameterizable distributions, making the training process differentiable. We learn their parameters in a Bayesian framework where a data-dependent prior is introduced for regularization. We apply the proposed stochastic attention modules to various attention-based models, with applications to graph node classification, visual question answering, image captioning, machine translation, and language understanding. Our experiments show the proposed method brings consistent improvements over the corresponding baselines.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 10; 2023: 10; 2024: 11; 2025: 6; 2026: 5.

51 papers.

Sort:

By area:

  1. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  2. 2021Adversarially Adaptive Normalization for Single Domain Generalizationcited by 85
  3. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 48
  4. 2022Visual Attention Methods in Deep Learning: An In-Depth Surveycited by 37
  5. 2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 35
  6. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  7. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  8. 2023POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained modelscited by 29
  9. 2023Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Modelscited by 29
  10. 2021Learning with Different Amounts of Annotation: From Zero to Many Labelscited by 28
  11. 2021Contextual Dropout: An Efficient Sample-Dependent Dropout Modulecited by 26
  12. 2021Bayesian Attention Belief Networkscited by 25
38 more
  1. 2021Sawtooth Factorial Topic Embeddings Guided Gamma Belief Networkcited by 23
  2. 2022ALLSH: Active Learning Guided by Local Sensitivity and Hardnesscited by 23
  3. 2022The Neural Process Family: Survey, Applications and Perspectivescited by 23
  4. 2023Calibrating Transformers via Sparse Gaussian Processescited by 21
  5. 2023Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systemscited by 19
  6. 2022Neural Processes with Stochastic Attention: Paying more attention to the context datasetcited by 16
  7. 2021Alignment Attention by Matching Key and Query Distributionscited by 14
  8. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learningcited by 14
  9. 2022A Unified Framework for Alternating Offline Model Training and Policy Learningcited by 14
  10. 2021Pathologies in priors and inference for Bayesian transformerscited by 13
  11. 2023Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Modelscited by 12
  12. 2024Training-Free Bayesianization for Low-Rank Adapters of Large Language Modelscited by 10
  13. 2021TopicNet: Semantic Graph-Guided Topic Discoverycited by 9
  14. 2025Revisiting Kernel Attention with Correlated Gaussian Process Representationcited by 9
  15. 2023Knowledge-aware Bayesian Co-attention for Multimodal Emotion Recognitioncited by 8
  16. 2022miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddingscited by 7
  17. 2022Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Modelscited by 6
  18. 2022Knowledge-Aware Bayesian Deep Topic Modelcited by 5
  19. 2024Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processescited by 4
  20. 2024Language Rectified Flow: Advancing Diffusion Language Generation with Probabilistic Flowscited by 4
  21. 2024Bayesian Sheaf Neural Networkscited by 4
  22. 2026UAT-LITE: Inference-Time Uncertainty-Aware Attention for Pretrained Transformerscited by 4
  23. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 3
  24. 2026Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformerscited by 3
  25. 2024Beyond Spectral Decomposition: Bayesian Contrastive Learning and its Non-negative Formulation via Factor Analysiscited by 2
  26. 2025Dropouts in Confidence: Moral Uncertainty in Human-LLM Alignmentcited by 2
  27. 2023Toward Robust Uncertainty Estimation with Random Activation Functionscited by 1
  28. 2024Scalable Weibull Graph Attention Autoencoder for Modeling Document Networkscited by 1
  29. 2025TPK: Trustworthy Trajectory Prediction Integrating Prior Knowledge For Interpretability and Kinematic Feasibilitycited by 1
  30. 2025A Free Probabilistic Framework for Analyzing the Transformer-based Language Modelscited by 1
  31. 2025Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problemscited by 1
  32. 2021Cascaded Head-colliding Attentioncited by 0
  33. 2023BayesSpeech: A Bayesian Transformer Network for Automatic Speech Recognitioncited by 0
  34. 2024An Attention-Based Deep Learning Architecture for Real-Time Monocular Visual Odometry: Applications to GPS-free Drone Navigationcited by 0
  35. 2024Switchable Decision: Dynamic Neural Generation Networkscited by 0
  36. 2024Bayesian Low-Rank LeArning (Bella): A Practical Approach to Bayesian Neural Networkscited by 0
  37. 2026Soft Specialists: α-R'enyi Ensembles for Uncertainty-Aware LLM Post-Trainingcited by 0
  38. 2026Information-Regularized Attention for Visual-Centric Reasoningcited by 0

50 most cited of 51 All 51 (JSON)