# ST-MoE-BERT: A Spatial-Temporal Mixture-of-Experts Framework for Long-Term Cross-City Mobility Prediction

arXiv: https://arxiv.org/abs/2410.14099 · HTML: https://mlsanity.com/paper/2410.14099/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in MoE for time series and graphs, one of 12 directions in Mixture-of-experts.
- Builds on 15 of its 42 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Placed in

- Mixture-of-experts / [MoE for time series and graphs](https://mlsanity.com/moe/moe-for-time-series-and-graphs/)

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution](https://mlsanity.com/paper/2306.15794/) (2023)
- [CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP](https://mlsanity.com/paper/2110.11316/) (2021)
- [A Survey on Deep Learning for Human Mobility](https://mlsanity.com/paper/2012.02825/) (2020)
- [FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets](https://mlsanity.com/paper/2310.04793/) (2023)
- [MobTCast: Leveraging Auxiliary Trajectory Forecasting for Human Mobility Prediction](https://mlsanity.com/paper/2110.01401/) (2021)
- [Exploring Large Language Models for Human Mobility Prediction under Public Events](https://mlsanity.com/paper/2311.17351/) (2023)
- [COLA: Cross-city Mobility Transformer for Human Trajectory Simulation](https://mlsanity.com/paper/2403.01801/) (2024)
- [Large Language Models are Zero-Shot Next Location Predictors](https://mlsanity.com/paper/2405.20962/) (2024)
- [Decoupled Alignment for Robust Plug-and-Play Adaptation](https://mlsanity.com/paper/2406.01514/) (2024)
- [GEO-BLEU: Similarity Measure for Geospatial Sequences](https://mlsanity.com/paper/2112.07144/) (2021)
- [Mixing Individual and Collective Behaviours to Predict Out-of-Routine Mobility](https://mlsanity.com/paper/2404.02740/) (2024)

## Built on by (most cited first, 7 of 7)

- [FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoning](https://mlsanity.com/paper/2601.19001/) (2026)
- [UniMove: A Unified Model for Multi-city Human Mobility Prediction](https://mlsanity.com/paper/2508.06986/) (2025)
- [Attention Sinks and Outliers in Attention Residuals](https://mlsanity.com/paper/2605.17887/) (2026)
- [Efficient Temporal Tokenization for Mobility Prediction with Large Language Models](https://mlsanity.com/paper/2507.14017/) (2025)
- [User Trajectory Prediction Unifying Global and Local Temporal Information](https://mlsanity.com/paper/2508.02161/) (2025)
- [RAST-MoE-RL: A Regime-Aware Spatio-Temporal MoE Framework for Deep Reinforcement Learning in Ride-Hailing](https://mlsanity.com/paper/2512.13727/) (2025)
- [Evolutionary Router Feature Generation for Zero-Shot Graph Anomaly Detection with Mixture-of-Experts](https://mlsanity.com/paper/2602.11622/) (2026)

## Abstract (arXiv)

Predicting human mobility across multiple cities presents significant challenges due to the complex and diverse spatial-temporal dynamics inherent in different urban environments. In this study, we propose a robust approach to predict human mobility patterns called ST-MoE-BERT. Compared to existing methods, our approach frames the prediction task as a spatial-temporal classification problem. Our methodology integrates the Mixture-of-Experts architecture with BERT model to capture complex mobility dynamics and perform the downstream human mobility prediction task. Additionally, transfer learning is integrated to solve the challenge of data scarcity in cross-city prediction. We demonstrate the effectiveness of the proposed model on GEO-BLEU and DTW, comparing it to several state-of-the-art methods. Notably, ST-MoE-BERT achieves an average improvement of 8.29%.