# TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting

arXiv: https://arxiv.org/abs/2510.25502 · HTML: https://mlsanity.com/paper/2510.25502/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 52 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Learning to (Learn at Test Time): RNNs with Expressive Hidden States](https://mlsanity.com/paper/2407.04620/) (2024)
- [ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels](https://mlsanity.com/paper/1910.13051/) (2019)
- [Titans: Learning to Memorize at Test Time](https://mlsanity.com/paper/2501.00663/) (2024)
- [xLSTM: Extended Long Short-Term Memory](https://mlsanity.com/paper/2405.04517/) (2024)
- [Monash Time Series Forecasting Archive](https://mlsanity.com/paper/2105.06643/) (2021)
- [GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation](https://mlsanity.com/paper/2410.10393/) (2024)
- [RWKV-7 "Goose" with Expressive Dynamic State Evolution](https://mlsanity.com/paper/2503.14456/) (2025)
- [Parallelizing Linear Recurrent Neural Nets Over Sequence Length](https://mlsanity.com/paper/1709.04057/) (2017)
- [TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning](https://mlsanity.com/paper/2505.23719/) (2025)
- [ForecastPFN: Synthetically-Trained Zero-Shot Forecasting](https://mlsanity.com/paper/2311.01933/) (2023)
- [Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues](https://mlsanity.com/paper/2411.12537/) (2024)
- [Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts](https://mlsanity.com/paper/2410.10469/) (2024)
- [fev-bench: A Realistic Benchmark for Time Series Forecasting](https://mlsanity.com/paper/2509.26468/) (2025)

## Built on by (most cited first, 17 of 17)

- [Toto 2.0: Time Series Forecasting Enters the Scaling Era](https://mlsanity.com/paper/2605.20119/) (2026)
- [Reverso: Efficient Time Series Foundation Models for Zero-shot Forecasting](https://mlsanity.com/paper/2602.17634/) (2026)
- [TiRex-2: Generalizing TiRex to Multivariate Data and Streaming](https://mlsanity.com/paper/2607.01204/) (2026)
- [Interventional Time Series Priors for Causal Foundation Models](https://mlsanity.com/paper/2603.11090/) (2026)
- [OATS: Online Data Augmentation for Time Series Foundation Models](https://mlsanity.com/paper/2601.19040/) (2026)
- [Post-Training in Time Series Foundation Models: A Unifying Framework](https://mlsanity.com/paper/2607.20002/) (2026)
- [EXAONE Finance 1.0: An Attention-free Time Series Foundation Model for Financial Time Series](https://mlsanity.com/paper/2609.04239/) (2026)
- [EIDOS: Latent-Space Predictive Learning for Time Series Foundation Models](https://mlsanity.com/paper/2602.14024/) (2026)
- [Zero-shot Multivariate Time Series Forecasting Using Tabular Prior Fitted Networks](https://mlsanity.com/paper/2604.08400/) (2026)
- [Mix, Don't Pick: Why Synthetic Corpus Composition Matters for Time Series Foundation Model Pretraining](https://mlsanity.com/paper/2606.09912/) (2026)
- [FinVerse: Financial Time-Series Benchmark](https://mlsanity.com/paper/2608.03259/) (2026)
- [TinyCast: Probabilistic Zero-Shot Forecasting with Computed Periodicity](https://mlsanity.com/paper/2608.15767/) (2026)
- [JointFM-0.1: A Foundation Model for Multi-Target Joint Distributional Prediction](https://mlsanity.com/paper/2603.20266/) (2026)
- [Feature to Dynamics: Feature-space to Autoregression strategy for Zero-shot Time Series Forecasting](https://mlsanity.com/paper/2606.01289/) (2026)
- [MACROCAST: A Vintage-Consistent Time Series Foundation Model for Real-Time Macroeconomic Forecasting](https://mlsanity.com/paper/2606.28670/) (2026)
- [Distillation of Synthetic Data for Time Series Foundation Models](https://mlsanity.com/paper/2609.09586/) (2026)
- [EXAONE Demand 1.0: A Time Series Foundation Model for Demand Forecasting](https://mlsanity.com/paper/2609.30880/) (2026)

## Abstract (arXiv)

Foundation models for zero-shot time series forecasting face challenges in efficient long-horizon prediction and reproducibility, with existing synthetic-only approaches underperforming on challenging benchmarks. This paper presents TempoPFN, a univariate time series foundation model based on linear Recurrent Neural Networks (RNNs) pre-trained exclusively on synthetic data. The model uses a GatedDeltaProduct architecture with state-weaving for fully parallelizable training across sequence lengths, eliminating the need for windowing or summarization techniques while maintaining robust temporal state-tracking. Our comprehensive synthetic data pipeline unifies diverse generators, including stochastic differential equations, Gaussian processes, and audio synthesis, with novel augmentations. In zero-shot evaluations on the Gift-Eval, fev-bench and Chronos-ZS benchmarks, TempoPFN achieves top-tier competitive performance, outperforming all existing synthetic-only approaches and surpassing the majority of models trained on real-world data, while being more efficient than existing baselines by leveraging fully parallelizable training and inference. We open-source our complete data generation pipeline and training code, providing a reproducible foundation for future research.