mlsanityData to 7 Oct 2026

Calibrating Transformers via Sparse Gaussian Processes

arXiv 2303.02444 ↗First posted 4 Mar 2023cs.LG stat.MLcited by 21

Not on the map. Why?

Abstract (arXiv)

Transformer models have achieved profound success in prediction tasks in a wide range of applications in natural language processing, speech recognition and computer vision. Extending Transformer's success to safety-critical domains requires calibrated uncertainty estimation which remains under-explored. To address this, we propose Sparse Gaussian Process attention (SGPA), which performs Bayesian inference directly in the output space of multi-head attention blocks (MHAs) in transformer to calibrate its uncertainty. It replaces the scaled dot-product operation with a valid symmetric kernel and uses sparse Gaussian processes (SGP) techniques to approximate the posterior processes of MHA outputs. Empirically, on a suite of prediction tasks on text, images and graphs, SGPA-based Transformers achieve competitive predictive accuracy, while noticeably improving both in-distribution calibration and out-of-distribution robustness and detection.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 38 of its 50 references that are in the map. First posted on arXiv 4 Mar 2023.

Builds on

38 of 50 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  8. 2017On Calibration of Modern Neural Networkscited by 4,860
  9. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  10. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  11. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  12. 2018Deep contextualized word representationscited by 2,285
26 more
  1. 2018Neural Network Acceptability Judgmentscited by 952
  2. 2020Benchmarking Graph Neural Networkscited by 672
  3. 2015Weight Uncertainty in Neural Networkscited by 554
  4. 2015Deep Kernel Learningcited by 440
  5. 2022DeiT III: Revenge of the ViTcited by 417
  6. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  7. 2013Gaussian Processes for Big Datacited by 377
  8. 2020Calibrating Deep Neural Networks using Focal Losscited by 353
  9. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 227
  10. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  11. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  12. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  13. 2017Doubly Stochastic Variational Inference for Deep Gaussian Processescited by 143
  14. 2020Efficiently Sampling Functions from Gaussian Process Posteriorscited by 125
  15. 2018Bayesian Layers: A Module for Neural Network Uncertaintycited by 57
  16. 2020Bayesian Attention Modulescited by 51
  17. 2017Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networkscited by 48
  18. 2021Bayesian Transformer Language Models for Speech Recognitioncited by 22
  19. 2017Variational Inference for Gaussian Process Models with Linear Complexitycited by 21
  20. 2018Orthogonally Decoupled Variational Gaussian Processescited by 19
  21. 2020Sparse within Sparse Gaussian Processes using Neighbor Informationcited by 18
  22. 2021Sparse Uncertainty Representation in Deep Learning with Inducing Weightscited by 16
  23. 2021Pathologies in priors and inference for Bayesian transformerscited by 13
  24. 2021Scalable Variational Gaussian Processes via Harmonic Kernel Decompositioncited by 8
  25. 2021Wide Mean-Field Variational Bayesian Neural Networks Ignore the Datacited by 7
  26. 2021Input Dependent Sparse Gaussian Processescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 5; 2026: 7.

21 papers.

Sort:

By area:

  1. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  2. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 48
  3. 2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 35
  4. 2023Understanding Emergent In-Context Learning from a Kernel Regression Perspectivecited by 23
  5. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
  6. 2025Revisiting Kernel Attention with Correlated Gaussian Process Representationcited by 9
  7. 2024Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysiscited by 7
  8. 2024Elliptical Attentioncited by 7
  9. 2024Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processescited by 4
  10. 2025Bayesian Mixture of Experts For Large Language Modelscited by 4
  11. 2026UAT-LITE: Inference-Time Uncertainty-Aware Attention for Pretrained Transformerscited by 4
  12. 2024A Strategy to Combine 1stGen Transformers and Open LLMs for Automatic Text Classificationcited by 3
9 more
  1. 2026Barriers to Discrete Reasoning with Transformers: A Survey Across Depth, Exactness, and Bandwidthcited by 3
  2. 2026Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformerscited by 3
  3. 2025h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objectivecited by 2
  4. 2025Compact Vision Transformer by Reduction of Kernel Complexitycited by 0
  5. 2025A decoupled alignment kernel for peptide membrane permeability predictionscited by 0
  6. 2026Diffusion-Inspired Reconfiguration of Transformers for Uncertainty Calibrationcited by 0
  7. 2026In-Context Function Learning in Large Language Modelscited by 0
  8. 2026Precision Tracked Transformer via Kalman Filtering, Kriging and Process Noisecited by 0
  9. 2026Interdomain Attention: Beyond Token-Level Key-Value Memorycited by 0