# SST: Multi-Scale Hybrid Mamba-Transformer Experts for Time Series Forecasting

arXiv: https://arxiv.org/abs/2404.14757 · HTML: https://mlsanity.com/paper/2404.14757/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Time series forecasting with SSMs, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 25 of its 50 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Time series forecasting with SSMs](https://mlsanity.com/ssm/time-series-forecasting-with-ssms/)

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [Jamba: A Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2403.19887/) (2024)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions](https://mlsanity.com/paper/1909.00590/) (2019)
- [Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2402.00789/) (2024)
- [Revisiting Spatial-Temporal Similarity: A Deep Learning Framework for Traffic Prediction](https://mlsanity.com/paper/1803.01254/) (2018)
- [TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecasting](https://mlsanity.com/paper/2403.09898/) (2024)
- [ChatGPT Informed Graph Neural Network for Stock Movement Prediction](https://mlsanity.com/paper/2306.03763/) (2023)
- [Multi-resolution Time-Series Transformer for Long-term Forecasting](https://mlsanity.com/paper/2311.04147/) (2023)
- [Weather2K: A Multivariate Spatio-Temporal Benchmark Dataset for Meteorological Forecasting Based on Real-Time Observation Data from Ground Weather Stations](https://mlsanity.com/paper/2302.10493/) (2023)
- [Uncovering Selective State Space Model's Capabilities in Lifelong Sequential Recommendation](https://mlsanity.com/paper/2403.16371/) (2024)
- [DeepTransport: Learning Spatial-Temporal Dependency for Traffic Condition Forecasting](https://mlsanity.com/paper/1709.09585/) (2017)

## Built on by (most cited first, 16 of 16)

- [A Survey of Mamba](https://mlsanity.com/paper/2408.01129/) (2024)
- [EHRMamba: Towards Generalizable and Scalable Foundation Models for Electronic Health Records](https://mlsanity.com/paper/2405.14567/) (2024)
- [MemoryMamba: Memory-Augmented State Space Model for Defect Recognition](https://mlsanity.com/paper/2405.03673/) (2024)
- [EchoMamba4Rec: Harmonizing Bidirectional State Space Models with Spectral Filtering for Advanced Sequential Recommendation](https://mlsanity.com/paper/2406.02638/) (2024)
- [Sequential Order-Robust Mamba for Time Series Forecasting](https://mlsanity.com/paper/2410.23356/) (2024)
- [FLDmamba: Integrating Fourier and Laplace Transform Decomposition with Mamba for Enhanced Time Series Prediction](https://mlsanity.com/paper/2507.12803/) (2025)
- [SS4Rec: Continuous-Time Sequential Recommendation with State Space Models](https://mlsanity.com/paper/2502.08132/) (2025)
- [Technologies on Effectiveness and Efficiency: A Survey of State Spaces Models](https://mlsanity.com/paper/2503.11224/) (2025)
- [Regularity and Stability Properties of Selective SSMs with Discontinuous Gating](https://mlsanity.com/paper/2505.11602/) (2025)
- [Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mamba](https://mlsanity.com/paper/2406.16722/) (2024)
- [STM3: Mixture of Multiscale Mamba for Long-Term Spatio-Temporal Time-Series Prediction](https://mlsanity.com/paper/2508.12247/) (2025)
- [PointNet4D: A Lightweight 4D Point Cloud Video Backbone for Online and Offline Perception in Robotic Applications](https://mlsanity.com/paper/2512.01383/) (2025)
- [M3: Mamba-assisted Multi-Circuit Optimization via MBRL with Effective Scheduling](https://mlsanity.com/paper/2411.16019/) (2024)
- [ChromFound: Towards A Universal Foundation Model for Single-Cell Chromatin Accessibility Data](https://mlsanity.com/paper/2505.12638/) (2025)
- [Memory in Deep Time-Series Models](https://mlsanity.com/paper/2609.06006/) (2026)
- [Correct then Forecast: Observer State-Space Models for Time Series Forecasting](https://mlsanity.com/paper/2609.33566/) (2026)

## Abstract (arXiv)

Time series forecasting has made significant advances, including with Transformer-based models. The attention mechanism in Transformer effectively captures temporal dependencies by attending to all past inputs simultaneously. However, its quadratic complexity with respect to sequence length limits the scalability for long-range modeling. Recent state space models (SSMs) such as Mamba offer a promising alternative by achieving linear complexity without attention. Yet, Mamba compresses historical information into a fixed-size latent state, potentially causing information loss and limiting representational effectiveness. This raises a key research question: Can we design a hybrid Mamba-Transformer architecture that is both effective and efficient for time series forecasting? To address it, we adapt a hybrid Mamba-Transformer architecture Mambaformer, originally proposed for language modeling, to the time series domain. Preliminary experiments reveal that naively stacking Mamba and Transformer layers in Mambaformer is suboptimal for time series forecasting, due to an information interference problem. To mitigate this issue, we introduce a new time series decomposition strategy that separates time series into long-range patterns and short-range variations. Then we show that Mamba excels at capturing long-term structures, while Transformer is more effective at modeling short-term dynamics. Building on this insight, we propose State Space Transformer (SST), a multi-scale hybrid model with expert modules: a Mamba expert for long-range patterns and a Transformer expert for short-term variations. SST also employs a multi-scale patching mechanism to adaptively adjust time series resolution: low resolution for long-term patterns and high resolution for short-term variations. Experiments show that SST obtains SOTA performance with linear scalability. The code is at https://github.com/XiongxiaoXu/SST.