mlsanityData to 7 Oct 2026

Spatial Mixture-of-Experts

arXiv 2211.13491 ↗First posted 24 Nov 2022cs.LGcited by 14

On the map: Mixture-of-experts → MoE for time series and graphs

Abstract (arXiv)

Many data have an underlying dependence on spatial location; it may be weather on the Earth, a simulation on a mesh, or a registered image. Yet this feature is rarely taken advantage of, and violates common assumptions made by many neural network layers, such as translation equivariance. Further, many works that do incorporate locality fail to capture fine-grained structure. To address this, we introduce the Spatial Mixture-of-Experts (SMoE) layer, a sparsely-gated layer that learns spatial structure in the input domain and routes experts at a fine-grained level to utilize it. We also develop new techniques to train SMoEs, including a self-supervised routing loss and damping expert errors. Finally, we show strong results for SMoEs on numerous tasks, and set new state-of-the-art results for medium-range weather prediction and post-processing ensemble weather forecasts.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in MoE for time series and graphs, one of 12 directions in Mixture-of-experts. Builds on 71 of its 116 references that are in the map. First posted on arXiv 24 Nov 2022.

Builds on

71 of 116 references.

References · 71

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  9. 2016Wide Residual Networkscited by 3,057
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  12. 2017Non-local Neural Networkscited by 2,290
38 more
  1. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  2. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  3. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,869
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  5. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  6. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  8. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  9. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 748
  10. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  11. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  12. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  13. 2017Residual Attention Network for Image Classificationcited by 385
  14. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  15. 2016Dynamic Filter Networkscited by 318
  16. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  17. 2021Skillful Precipitation Nowcasting using Deep Generative Models of Radarcited by 308
  18. 2018An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solutioncited by 302
  19. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  20. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  21. 2020WeatherBench: A benchmark dataset for data-driven weather forecastingcited by 272
  22. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  23. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  24. 2015Conditional Random Fields as Recurrent Neural Networkscited by 213
  25. 2017Deep Learning for Precipitation Nowcasting: A Benchmark and A New Modelcited by 180
  26. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  27. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  28. 2022Forecasting Global Weather with Graph Neural Networkscited by 165
  29. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  30. 2020MetNet: A Neural Weather Model for Precipitation Forecastingcited by 128
  31. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 125
  32. 2020Dynamic Model Pruning with Feedbackcited by 125
  33. 2018Graph-Based Global Reasoning Networkscited by 120
  34. 2018Attention Branch Network: Learning of Attention Mechanism for Visual Explanationcited by 108
  35. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  36. 2016Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesiscited by 102
  37. 2018Neural networks for post-processing ensemble weather forecastscited by 95
  38. 2013Deep Learning of Representations: Looking Forwardcited by 84

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 4; 2026: 3.

14 papers.

Sort:

By area:

  1. 2023Mixture-of-Linear-Experts for Long-term Time Series Forecastingcited by 29
  2. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  3. 2024TESTAM: A Time-Enhanced Spatio-Temporal Attention Model with Mixture of Expertscited by 18
  4. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  5. 2023AdapTraj: A Multi-Source Domain Generalization Framework for Multi-Agent Trajectory Predictioncited by 5
  6. 2024Graph Knowledge Distillation to Mixture of Expertscited by 2
  7. 2024A Time Series is Worth Five Experts: Heterogeneous Mixture of Experts for Traffic Flow Predictioncited by 2
  8. 2025Knowledge-Guided Adaptive Mixture of Experts for Precipitation Predictioncited by 1
  9. 2025Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptationcited by 0
  10. 2025SHMoAReg: Spark Deformable Image Registration via Spatial Heterogeneous Mixture of Experts and Attention Headscited by 0
  11. 2025RadioKMoE: Knowledge-Guided Radiomap Estimation with Kolmogorov-Arnold Networks and Mixture-of-Expertscited by 0
  12. 2026Rule-Based Spatial Mixture-of-Experts U-Net for Explainable Edge Detectioncited by 0
2 more
  1. 2026AdaWeather: Adaptively Mixing Probabilistic Weather Forecasts with Logarithmic Regretcited by 0
  2. 2026When Big Data Becomes a Curse: Spatial Heterogeneity and the Limits of Learning from Passive Acoustic Monitoring Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024HiMoE: Heterogeneity-Informed Mixture-of-Experts for Fair Spatial-Temporal Forecasting
  2. 2025Generating ensembles of spatially-coherent in-situ forecasts using flow matching
  3. 2025MoWE : A Mixture of Weather Experts
  4. 2025Rethinking Sparse Mixture of Experts from a Unified Perspective
  5. 2026Eigenvectors of Experts are Training-free Non-collapsing Routers
  6. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy
  7. 2026Structure Aware Neural Architecture Search for Mixture of Experts
  8. 2025STCast: Adaptive Boundary Alignment for Global and Regional Weather Forecasting