mlsanityData to 14 Oct 2026

ORBIT-2: Scaling Exascale Vision Foundation Models for Weather and Climate Downscaling

arXiv 2505.04802 ↗First posted 7 May 2025cs.LG astro-ph.EP cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Sparse observations and coarse-resolution climate models limit effective regional decision-making, underscoring the need for robust downscaling. However, existing AI methods struggle with generalization across variables and geographies and are constrained by the quadratic complexity of Vision Transformer (ViT) self-attention. We introduce ORBIT-2, a scalable foundation model for global, hyper-resolution climate downscaling. ORBIT-2 incorporates two key innovations: (1) Residual Slim ViT (Reslim), a lightweight architecture with residual learning and Bayesian regularization for efficient, robust prediction; and (2) TILES, a tile-wise sequence scaling algorithm that reduces self-attention complexity from quadratic to linear, enabling long-sequence processing and massive parallelism. ORBIT-2 scales to 10 billion parameters across 65,536 GPUs, achieving up to 4.1 exaFLOPS sustained throughput and 74--98% strong scaling efficiency. It supports downscaling to 0.9 km global resolution and processes sequences up to 4.2 billion tokens. On 7 km resolution benchmarks, ORBIT-2 achieves high accuracy with R2 scores in the range of 0.98--0.99 against observational data.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 23 of its 55 references that are in the map. First posted on arXiv 7 May 2025.

Builds on

23 of 55 references.

References · 23

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  5. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  6. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  7. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  8. 2023ClimaX: A foundation model for weather and climatecited by 337
  9. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  10. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 171
  11. 2021Transformer for Single Image Super-Resolutioncited by 150
  12. 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 90
11 more
  1. 2023ClimateLearn: Benchmarking Machine Learning for Weather and Climate Modelingcited by 47
  2. 2024Prithvi WxC: Foundation Model for Weather and Climatecited by 31
  3. 2020torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Modelscited by 26
  4. 2021PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformerscited by 23
  5. 2024Conditional diffusion models for downscaling and bias correction of Earth system model precipitationcited by 17
  6. 2024ORBIT: Oak Ridge Base Foundation Model for Earth System Predictabilitycited by 17
  7. 2023Algorithmic Hallucinations of Near-Surface Winds: Statistical Downscaling with Generative Adversarial Networks to Convection-Permitting Scalescited by 13
  8. 2020Deep-learning based down-scaling of summer monsoon rainfall data over Indian regioncited by 10
  9. 2024Adaptive Patching for High-resolution Image Segmentation with Transformerscited by 6
  10. 2023Ultra-Long Sequence Distributed Transformercited by 3
  11. 2024Sequence Length Scaling in Vision Transformers for Scientific Images on Frontiercited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026On Neural Scaling Laws for Weather Emulation through Continual Trainingcited by 2
  2. 2026Global Attention with Linear Complexity for Exascale Generative Data Assimilation in Earth System Predictioncited by 2
  3. 2026Towards Scaling Law Analysis For Spatiotemporal Weather Datacited by 0
  4. 2026Transforming the Use of Earth Observation Data: Exascale Training of a Generative Compression Model with Historical Priors for up to 10,000x Data Reductioncited by 0
  5. 2026Less Tokens, Better Forecasts: Sparse Residual Routing for Efficient Weather Predictioncited by 0
  6. 2026TERRA: A Hierarchical Parallel Training and Memory Orchestration Framework for High-Resolution AI-based Earth Modelingcited by 0
  7. 2026Precipitation Downscaling Using Foundation Model-Conditioned Diffusioncited by 0
  8. 20264D Parallelism Unlocks Exascale Bayesian Neural Networks for High-Fidelity Atmospheric Modelingcited by 0
  9. 2026Structure-Guided Masked Autoencoders for Ultra-High Resolution Scientific Image Understandingcited by 0
  10. 2026Scalable In-Domain Self-Supervised Foundation Model for Dense Representation Transfer in High-Resolution Plant Imagingcited by 0