# Decoupled-Value Attention for Prior-Data Fitted Networks: GP Inference for Physical Equations

arXiv: https://arxiv.org/abs/2509.20950 · HTML: https://mlsanity.com/paper/2509.20950/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Random Feature Attention](https://mlsanity.com/paper/2103.02143/) (2021)
- [Tackling Climate Change with Machine Learning](https://mlsanity.com/paper/1906.05433/) (2019)
- [Transformers as Algorithms: Generalization and Stability in In-context Learning](https://mlsanity.com/paper/2301.07067/) (2023)
- [Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel](https://mlsanity.com/paper/1908.11775/) (2019)
- [Statistical Foundations of Prior-Data Fitted Networks](https://mlsanity.com/paper/2305.11097/) (2023)
- [Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks](https://mlsanity.com/paper/2310.20447/) (2023)
- [Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization](https://mlsanity.com/paper/2402.02746/) (2024)
- [PowerModels.jl: An Open-Source Framework for Exploring Power Flow Formulations](https://mlsanity.com/paper/1711.01728/) (2017)
- [Revisiting Kernel Attention with Correlated Gaussian Process Representation](https://mlsanity.com/paper/2502.20525/) (2025)
- [Learning "best" kernels from data in Gaussian process regression. With application to aerodynamics](https://mlsanity.com/paper/2206.02563/) (2022)
- [Risk-Aware Control and Optimization for High-Renewable Power Grids](https://mlsanity.com/paper/2204.00950/) (2022)
- [Prior-Fitted Networks Scale to Larger Datasets When Treated as Weak Learners](https://mlsanity.com/paper/2503.01256/) (2025)
- [How Good are Low-Rank Approximations in Gaussian Process Regression?](https://mlsanity.com/paper/2004.01584/) (2020)
- [Gaussian Processes in Power Systems: Techniques, Applications, and Future Works](https://mlsanity.com/paper/2505.15950/) (2025)

## Built on by (most cited first, 1 of 1)

- [Mechanistic Evidence for Spectral Structures in Prior-Data Fitted Networks](https://mlsanity.com/paper/2601.21731/) (2026)

## Abstract (arXiv)

Prior-data fitted networks (PFNs) are a promising alternative to time-consuming Gaussian process (GP) inference for creating fast surrogates of physical systems. PFN reduces the computational burden of GP-training by replacing Bayesian inference in GP with a single forward pass of a learned prediction model. However, with standard Transformer attention, PFNs show limited effectiveness on high-dimensional regression tasks. We introduce Decoupled-Value Attention (DVA)-- motivated by the GP property that the function space is fully characterized by the kernel over inputs and the predictive mean is a weighted sum of training targets. DVA computes similarities from inputs only and propagates labels solely through values. Thus, the proposed DVA mirrors the GP update while remaining kernel-free. We demonstrate that PFNs are backbone architecture invariant and the crucial factor for scaling PFNs is the attention rule rather than the architecture itself. Specifically, our results demonstrate that (a) localized attention consistently reduces out-of-sample validation loss in PFNs across different dimensional settings, with validation loss reduced by more than 50% in five- and ten-dimensional cases, and (b) the role of attention is more decisive than the choice of backbone architecture, showing that CNN, RNN and LSTM-based PFNs can perform at par with their Transformer-based counterparts. The proposed PFNs provide 64-dimensional power flow equation approximations with a mean absolute error of the order of E-03, while being over 80x faster than exact GP inference.