# Field-Space Attention for Structure-Preserving Earth System Transformers

arXiv: https://arxiv.org/abs/2512.20350 · HTML: https://mlsanity.com/paper/2512.20350/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 44 references that are in the map.
- First posted on arXiv 23 Dec 2025.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Neural General Circulation Models for Weather and Climate](https://mlsanity.com/paper/2311.07222/) (2023)
- [Residual Corrective Diffusion Modeling for Km-scale Atmospheric Downscaling](https://mlsanity.com/paper/2309.15214/) (2023)
- [FourCastNet 3: A geometric approach to probabilistic machine-learning weather forecasting at scale](https://mlsanity.com/paper/2507.12144/) (2025)
- [AtmoRep: A stochastic model of atmosphere dynamics using large scale representation learning](https://mlsanity.com/paper/2308.13280/) (2023)
- [Will Artificial Intelligence supersede Earth System and Climate Models?](https://mlsanity.com/paper/2101.09126/) (2021)
- [On the Foundations of Earth and Climate Foundation Models](https://mlsanity.com/paper/2405.04285/) (2024)
- [Hard-Constrained Deep Learning for Climate Downscaling](https://mlsanity.com/paper/2208.05424/) (2022)
- [Theoretical Analysis of Inductive Biases in Deep Convolutional Networks](https://mlsanity.com/paper/2305.08404/) (2023)
- [HEAL-SWIN: A Vision Transformer On The Sphere](https://mlsanity.com/paper/2307.07313/) (2023)
- [Enhancing Performance of Vision Transformers on Small Datasets through Local Inductive Bias Incorporation](https://mlsanity.com/paper/2305.08551/) (2023)
- [Dynamic Deep Learning Based Super-Resolution For The Shallow Water Equations](https://mlsanity.com/paper/2404.06400/) (2024)

## Built on by (most cited first, 1 of 1)

- [Field-Space Autoencoder for Scalable Climate Emulators](https://mlsanity.com/paper/2601.15102/) (2026)

## Abstract (arXiv)

Accurate and physically consistent modeling of Earth system dynamics requires machine-learning architectures that operate directly on continuous geophysical fields and preserve their underlying geometric structure. Here we introduce Field-Space attention, a mechanism for Earth system Transformers that computes attention in the physical domain rather than in a learned latent space. By maintaining all intermediate representations as continuous fields on the sphere, the architecture enables interpretable internal states and facilitates the enforcement of scientific constraints. The model employs a fixed, non-learned multiscale decomposition and learns structure-preserving deformations of the input field, allowing coherent integration of coarse and fine-scale information while avoiding the optimization instabilities characteristic of standard single-scale Vision Transformers. Applied to global temperature super-resolution on a HEALPix grid, Field-Space Transformers converge more rapidly and stably than conventional Vision Transformers and U-Net baselines, while requiring substantially fewer parameters. The explicit preservation of field structure throughout the network allows physical and statistical priors to be embedded directly into the architecture, yielding improved fidelity and reliability in data-driven Earth system modeling. These results position Field-Space Attention as a compact, interpretable, and physically grounded building block for next-generation Earth system prediction and generative modeling frameworks.