# TESTAM: A Time-Enhanced Spatio-Temporal Attention Model with Mixture of Experts

arXiv: https://arxiv.org/abs/2403.02600 · HTML: https://mlsanity.com/paper/2403.02600/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in MoE for time series and graphs, one of 12 directions in Mixture-of-experts.
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 5 Mar 2024.

## Placed in

- Mixture-of-experts / [MoE for time series and graphs](https://mlsanity.com/moe/moe-for-time-series-and-graphs/)

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting](https://mlsanity.com/paper/1709.04875/) (2017)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Graph WaveNet for Deep Spatial-Temporal Graph Modeling](https://mlsanity.com/paper/1906.00121/) (2019)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks](https://mlsanity.com/paper/2005.11650/) (2020)
- [Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting](https://mlsanity.com/paper/2007.02842/) (2020)
- [GMAN: A Graph Multi-Attention Network for Traffic Prediction](https://mlsanity.com/paper/1911.08415/) (2019)
- [Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting](https://mlsanity.com/paper/2103.07719/) (2021)
- [Learning Factored Representations in a Deep Mixture of Experts](https://mlsanity.com/paper/1312.4314/) (2013)
- [Time2Vec: Learning a Vector Representation of Time](https://mlsanity.com/paper/1907.05321/) (2019)
- [Spatial-Temporal Transformer Networks for Traffic Flow Forecasting](https://mlsanity.com/paper/2001.02908/) (2020)
- [Discrete Graph Structure Learning for Forecasting Multiple Time Series](https://mlsanity.com/paper/2101.06861/) (2021)
- [Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning](https://mlsanity.com/paper/1711.01239/) (2017)
- [Spatio-Temporal Meta-Graph Learning for Traffic Forecasting](https://mlsanity.com/paper/2211.14701/) (2022)
- [Coupled Layer-wise Graph Convolution for Transportation Demand Prediction](https://mlsanity.com/paper/2012.08080/) (2020)
- [Deciding How to Decide: Dynamic Routing in Artificial Neural Networks](https://mlsanity.com/paper/1703.06217/) (2017)
- [Learning to Remember Patterns: Pattern Matching Memory Networks for Traffic Forecasting](https://mlsanity.com/paper/2110.10380/) (2021)
- [Spatial Mixture-of-Experts](https://mlsanity.com/paper/2211.13491/) (2022)
- [A Visual Analytics System for Improving Attention-based Traffic Forecasting Models](https://mlsanity.com/paper/2208.04350/) (2022)
- [At a Glance: Pixel Approximate Entropy as a Measure of Line Chart Complexity](https://mlsanity.com/paper/1811.03180/) (2018)

## Built on by (most cited first, 18 of 18)

- [Heterogeneity-Informed Meta-Parameter Learning for Spatiotemporal Time Series Forecasting](https://mlsanity.com/paper/2405.10800/) (2024)
- [FACTS: A Factored State-Space Framework For World Modelling](https://mlsanity.com/paper/2410.20922/) (2024)
- [How Different from the Past? Spatio-Temporal Time Series Forecasting with Self-Supervised Deviation Learning](https://mlsanity.com/paper/2510.04908/) (2025)
- [Navigating Spatio-Temporal Heterogeneity: A Graph Transformer Approach for Traffic Forecasting](https://mlsanity.com/paper/2408.10822/) (2024)
- [MoGERNN: An Inductive Traffic Predictor for Unobserved Locations](https://mlsanity.com/paper/2501.12281/) (2025)
- [Memory-enhanced Invariant Prompt Learning for Urban Flow Prediction under Distribution Shifts](https://mlsanity.com/paper/2412.05534/) (2024)
- [ST-LINK: Spatially-Aware Large Language Models for Spatio-Temporal Forecasting](https://mlsanity.com/paper/2509.13753/) (2025)
- [HieraMix: A Hierarchical MLP-Mixer for Large-Scale Traffic Forecasting](https://mlsanity.com/paper/2512.07854/) (2025)
- [Spatio-Temporal Partial Sensing Forecast for Long-term Traffic](https://mlsanity.com/paper/2408.02689/) (2024)
- [HiMoE: Heterogeneity-Informed Mixture-of-Experts for Fair Spatial-Temporal Forecasting](https://mlsanity.com/paper/2412.00316/) (2024)
- [DRAN: A Distribution and Relation Adaptive Network for Spatio-temporal Forecasting](https://mlsanity.com/paper/2504.01531/) (2025)
- [Spatio-temporal Multivariate Time Series Forecast with Chosen Variables](https://mlsanity.com/paper/2510.24027/) (2025)
- [Towards Resilient Transportation: A Conditional Transformer for Accident-Informed Traffic Forecasting](https://mlsanity.com/paper/2512.09398/) (2025)
- [MixTTE: Multi-Level Mixture-of-Experts for Scalable and Adaptive Travel Time Estimation](https://mlsanity.com/paper/2601.02943/) (2026)
- [TSNN: A Non-parametric and Interpretable Framework for Traffic Time Series Forecasting](https://mlsanity.com/paper/2605.09208/) (2026)
- [Graph-Conditioned Mixture of Graph Neural Network Experts for Traffic Forecasting](https://mlsanity.com/paper/2605.30486/) (2026)
- [Frequency-Domain Multi-Modality Transportation Modeling](https://mlsanity.com/paper/2607.08475/) (2026)
- [Scene-Conditioned Relation Routing for urban cellular activity forecasting](https://mlsanity.com/paper/2609.20209/) (2026)

## Abstract (arXiv)

Accurate traffic forecasting is challenging due to the complex dependency on road networks, various types of roads, and the abrupt speed change due to the events. Recent works mainly focus on dynamic spatial modeling with adaptive graph embedding or graph attention having less consideration for temporal characteristics and in-situ modeling. In this paper, we propose a novel deep learning model named TESTAM, which individually models recurring and non-recurring traffic patterns by a mixture-of-experts model with three experts on temporal modeling, spatio-temporal modeling with static graph, and dynamic spatio-temporal dependency modeling with dynamic graph. By introducing different experts and properly routing them, TESTAM could better model various circumstances, including spatially isolated nodes, highly related nodes, and recurring and non-recurring events. For the proper routing, we reformulate a gating problem into a classification problem with pseudo labels. Experimental results on three public traffic network datasets, METR-LA, PEMS-BAY, and EXPY-TKY, demonstrate that TESTAM achieves a better indication and modeling of recurring and non-recurring traffic. We published the official code at https://github.com/HyunWookL/TESTAM