mlsanityData to 14 Oct 2026

(Sparse) Attention to the Details: Preserving Spectral Fidelity in ML-based Weather Forecasting Models

arXiv 2604.16429 ↗First posted 6 Apr 2026cs.LG cs.AI cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

We introduce Mosaic, a probabilistic weather forecasting model that addresses three failure modes of spectral degradation in ML-based weather prediction: spectral damping (statistical), high-frequency aliasing (architectural), and residual high-frequency leakage (parametric). Mosaic generates ensemble members through learned functional perturbations and operates on native-resolution grids via mesh-aligned block-sparse attention, a hardware-aligned mechanism that captures long-range dependencies at linear cost by sharing keys and values across spatially adjacent queries. At 1.5deg resolution with 214M parameters, Mosaic matches or outperforms models trained on 6× finer resolution on key variables and achieves state-of-the-art results among 1.5deg models, producing well-calibrated ensembles whose individual members exhibit near-perfect spectral alignment across all resolved frequencies. A 24-member, 10-day forecast takes under 12s on a single H100~GPU. Code is available at https://github.com/maxxxzdn/mosaic.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 37 of its 63 references that are in the map. First posted on arXiv 6 Apr 2026.

Builds on

37 of 63 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2020GLU Variants Improve Transformercited by 1,804
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  7. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  8. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  9. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  10. 2022Message Passing Neural PDE Solverscited by 386
  11. 2024Transolver: A Fast Transformer Solver for PDEs on General Geometriescited by 326
  12. 2023Neural General Circulation Models for Weather and Climatecited by 239
25 more
  1. 2023WeatherBench 2: A benchmark for the next generation of data-driven global weather modelscited by 236
  2. 2024Rotary Position Embedding for Vision Transformercited by 236
  3. 2023PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solverscited by 216
  4. 2023GenCast: Diffusion-based ensemble forecasting for medium-range weathercited by 177
  5. 2023Scaling transformer neural networks for skillful and reliable medium-range weather forecastingcited by 110
  6. 2025Skillful joint probabilistic weather forecasting from marginalscited by 63
  7. 2025FourCastNet 3: A geometric approach to probabilistic machine-learning weather forecasting at scalecited by 62
  8. 2023Towards Stability of Autoregressive Neural Operatorscited by 59
  9. 2024Integrating Neural Operators with Diffusion Models Improves Spectral Representation in Turbulence Modelingcited by 36
  10. 2024ArchesWeather & ArchesWeatherGen: a deterministic and generative model for efficient ML weather forecastingcited by 32
  11. 2025PDE-Transformer: Efficient and Versatile Transformers for Physics Simulationscited by 32
  12. 2023Advancing Parsimonious Deep Learning Weather Prediction using the HEALPix Meshcited by 28
  13. 2025Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Functioncited by 28
  14. 2025Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And Morecited by 28
  15. 2025Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systemscited by 22
  16. 2024Grounding Continuous Representations in Geometry: Equivariant Neural Fieldscited by 19
  17. 2025BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generationcited by 13
  18. 2025ProxyAttn: Guided Sparse Attention via Representative Headscited by 9
  19. 2025High-Resolution Probabilistic Data-Driven Weather Modeling with a Stretched-Gridcited by 9
  20. 2024HEAL-ViT: Vision Transformers on a spherical mesh for medium-range weather forecastingcited by 8
  21. 2024Leveraging data-driven weather models for improving numerical weather prediction skill through large-scale spectral nudgingcited by 8
  22. 2025PEAR: Equal Area Weather Forecasting on the Spherecited by 5
  23. 2025MAUSAM: An Observations-focused assessment of Global AI Weather Prediction Models During the South Asian Monsooncited by 4
  24. 2025Alias-Free ViT: Fractional Shift Invariance via Linear Attentioncited by 2
  25. 2025BSA: Ball Sparse Attention for Large-scale Geometriescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026On the sensitivity of machine-learned probabilistic weather forecast models to scale-aware scoring rulescited by 1
  2. 2026Lightweight Probabilistic Downscaling from a Deterministic Base Modelcited by 0
  3. 2026Proper Scoring Rule-based Diffusion for Probabilistic Weather Forecastingcited by 0
  4. 2026SO(3)-RoPE for Spherical Transformerscited by 0
  5. 2026Cross-Domain Pretraining for Steady-State Neural CFD Surrogatescited by 0