# FieldFormer: Locality-Aware Transformers for Spatio-Temporal Modeling on Sparse Sensor Networks

arXiv: https://arxiv.org/abs/2510.03589 · HTML: https://mlsanity.com/paper/2510.03589/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 4 Oct 2025.

## Builds on (most cited first, 23 of 23)

- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Graph WaveNet for Deep Spatial-Temporal Graph Modeling](https://mlsanity.com/paper/1906.00121/) (2019)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [Machine learning accelerated computational fluid dynamics](https://mlsanity.com/paper/2102.01010/) (2021)
- [Are Transformers universal approximators of sequence-to-sequence functions?](https://mlsanity.com/paper/1912.10077/) (2019)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [U-NO: U-shaped Neural Operators](https://mlsanity.com/paper/2204.11127/) (2022)
- [Spatial-Temporal Transformer Networks for Traffic Flow Forecasting](https://mlsanity.com/paper/2001.02908/) (2020)
- [Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data](https://mlsanity.com/paper/2206.06577/) (2022)
- [ImputeFormer: Low Rankness-Induced Transformers for Generalizable Spatiotemporal Imputation](https://mlsanity.com/paper/2312.01728/) (2023)
- [Transformers are Graph Neural Networks](https://mlsanity.com/paper/2506.22084/) (2025)
- [Physics Informed Token Transformer for Solving Partial Differential Equations](https://mlsanity.com/paper/2305.08757/) (2023)
- [On structural and practical identifiability](https://mlsanity.com/paper/2102.05100/) (2021)
- [RecFNO: a resolution-invariant flow and heat field reconstruction method from sparse observations via Fourier neural operator](https://mlsanity.com/paper/2302.09808/) (2023)
- [Physics-Informed Graph Neural Network for Spatial-temporal Production Forecasting](https://mlsanity.com/paper/2209.11885/) (2022)
- [Higher-order Spatio-temporal Physics-incorporated Graph Neural Network for Multivariate Time Series Imputation](https://mlsanity.com/paper/2405.10995/) (2024)
- [Structure is information: structural identifiability mappings for machine learning with partially observed dynamical systems](https://mlsanity.com/paper/2502.04131/) (2025)
- [Comprehensive Monitoring of Air Pollution Hotspots Using Sparse Sensor Networks](https://mlsanity.com/paper/2410.04309/) (2024)

## Built on by (most cited first, 1 of 1)

- [Spatial Context or Memory? Sizing the Inputs of Neural Surrogates for Partially Observed Fields](https://mlsanity.com/paper/2610.12723/) (2026)

## Abstract (arXiv)

Spatio-temporal sensor data in real-world systems is often sparse, noisy, and irregular, making latent field reconstruction fundamentally underconstrained. Under extreme sparsity, multiple physically plausible fields may remain consistent with the same observations, requiring models to rely on inductive biases about locality, transport, and spatial regularity. In such regimes, reliable reconstruction is concentrated around the observational support induced by the sensor network, making sensor-space modeling a more identifiable objective than unconstrained global field recovery. We introduce FieldFormer, a mesh-free transformer architecture for locality-aware sensor-space modeling in persistent sensor networks. For each query, FieldFormer aggregates local evidence using learnable velocity-scaled offsets that adapt neighborhood geometry to spatio-temporal dependencies. Neighborhoods are constructed as fixed maximal sparse contexts over nearby sensors and bounded temporal windows, enabling stable and scalable inference under extreme sparsity. A local transformer encoder integrates neighborhood information, while a coordinate-based neural field formulation supports mesh-free prediction. We evaluate FieldFormer on five synthetic and real-world benchmarks, including anisotropic heat diffusion, shallow-water dynamics, atmospheric transport, and pollution monitoring datasets. Results show that locality-aware reconstruction provides strong advantages when local domains of dependence remain observed, enabling FieldFormer to consistently outperform state-of-the-art baselines on sparse sensor-space prediction tasks.