mlsanityData to 14 Oct 2026

Raw-Routed Mixture of Adapters: A Causal Intervention for Routing Collapse in Time Series Foundation Models

arXiv 2609.39445 ↗First posted 30 Sep 2026cs.LG stat.MLcited by 0

On the map: Mixture-of-experts → MoE for time series and graphs

Abstract (arXiv)

Time series foundation models (TSFMs) commonly adapt to new data by attaching a single trainable head to a frozen backbone, a one-size-fits-all setup that underfits heterogeneous regimes. Replacing the head with a mixture of experts is the standard upgrade, but on instance-normalized backbones (the dominant TSFM design class) it fails: routing entropy collapses to zero and one expert absorbs every input, a failure we call normalization-induced routing collapse. Standard MoE rescue mechanisms do not repair it, because the cause is in the router's input, not its optimization. Pre-encoder normalization strips the statistics a router would need to tell regimes apart. A mutual-information decomposition makes this precise and yields a signal-ratio that, computed before training, predicts dataset vulnerability (Spearman ρ = -0.88). Eight causal controls, including a vision-modality replication, isolate instance normalization as the cause. The prescription is a minimal causal intervention: Raw-Routed Mixture of Adapters (RR-MoA), which routes on the raw, pre-normalization input. Under a strictly frozen backbone, RR-MoA wins 54/54 comparisons against the strongest fixed adapter and significantly outperforms LoRA, TRACE, AdaMix, and full fine-tuning. The effect generalizes across six backbones and an imputation task. Frozen RR-MoA also beats full fine-tuning by 12-79% (the Frozen Paradox); two architecturally distinct variants confirm the principle generalizes beyond this specific router.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE for time series and graphs, one of 12 directions in Mixture-of-experts. Builds on 78 of its 90 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

78 of 90 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scaling5 Mar
  2. 2024Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts14 Oct
  3. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts24 Sep

Other references · 75

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  8. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  9. 2024Mixtral of Expertscited by 1,823
  10. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  12. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
38 more
  1. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  2. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 931
  3. 2016Deep Variational Information Bottleneckcited by 930
  4. 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 879
  5. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  6. 2024Chronos: Learning the Language of Time Seriescited by 759
  7. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  8. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
  9. 2023A decoder-only foundation model for time-series forecastingcited by 692
  10. 2018MINE: Mutual Information Neural Estimationcited by 683
  11. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 632
  12. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  13. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 614
  14. 2019On Variational Bounds of Mutual Informationcited by 559
  15. 2022Transformers in Time Series: A Surveycited by 497
  16. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 482
  17. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  18. 2022Non-stationary Transformers: Exploring the Stationarity in Time Series Forecastingcited by 391
  19. 2021Monash Time Series Forecasting Archivecited by 267
  20. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  21. 2023From Sparse to Soft Mixtures of Expertscited by 248
  22. 2023TSMixer: An All-MLP Architecture for Time Series Forecastingcited by 240
  23. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  24. 2024GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluationcited by 212
  25. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  26. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 208
  27. 2023FITS: Modeling Time Series with 10k Parameterscited by 195
  28. 2024Mixture of LoRA Expertscited by 181
  29. 2025Sundial: A Family of Highly Capable Time Series Foundation Modelscited by 180
  30. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
  31. 2023UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecastingcited by 144
  32. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  33. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
  34. 2024TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysiscited by 116
  35. 2023Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecastingcited by 113
  36. 2024UniTS: A Unified Multi-Task Time Series Modelcited by 94
  37. 2024Timer-XL: Long-Context Transformers for Unified Time Series Forecastingcited by 87
  38. 2025This Time is Different: An Observability Perspective on Time Series Foundation Modelscited by 68

50 most cited of 75 All 75 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Universal Redundancies in Time Series Foundation Models
  2. 2023Causal normalizing flows: from theory to practice
  3. 2026Diversified Scaling Inference in Time Series Foundation Models
  4. 2026AME-TS: Anchored Mixture-of-Experts for Time Series Forecasting
  5. 2026When, Not How Much: Evaluating Time-Series Foundation Models on Sparse Events
  6. 2026TimeRouter: Efficient and Adaptive Routing of Time-Series Foundation Models
  7. 2026GRAM: Correcting Frozen Time-Series Foundation Models via Graph-Retrieved Amplitude Memory
  8. 2026When Does Context Routing Help? A Systematic Study of Multi-Modal Fusion in Time Series Forecasting