# Random Feature Gaussian Process Attention: Linear-Time Probabilistic Attention with Calibrated Uncertainty

arXiv: https://arxiv.org/abs/2610.08578 · HTML: https://mlsanity.com/paper/2610.08578/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 2 of its 33 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Builds on (most cited first, 2 of 2)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)

## Abstract (arXiv)

Transformers provide a state-of-the-art modeling framework, yet poor calibration limits their reliability in safety-critical applications. A promising direction addresses this issue by interpreting attention as a Gaussian process (GP) posterior, which enables principled uncertainty calibration but incurs cubic complexity in sequence length due to the inversion of the kernel; although decoupled GP variants reduced the cost to quadratic, the computation remains prohibitive in practice. In this paper, we propose the plug-and-play random Fourier feature Gaussian process attention (RFF-GPA) module, which represents the attention as a GP with a stationary kernel approximated by random Fourier features. This low-rank approximation results in linear-time complexity for approximating the posterior mean and variance, making it far more scalable compared to previous work. Empirical results on multiple real-world datasets show that our attention module improves calibration while maintaining predictive accuracy, and simultaneously reduces computational complexity to linear in the sequence length.