mlsanityData to 14 Oct 2026

Exploring the Potential of Probabilistic Transformer for Time Series Modeling: A Report on the ST-PT Framework

arXiv 2604.26762 ↗First posted 29 Apr 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The Probabilistic Transformer (PT) establishes that the Transformer's self-attention plus its feed-forward block is mathematically equivalent to Mean-Field Variational Inference (MFVI) on a Conditional Random Field (CRF). Under this equivalence the Transformer ceases to be a black-box neural network and becomes a programmable factor graph: graph topology, factor potentials, and the message-passing schedule are all explicit and inspectable primitives that can be engineered. PT was originally developed for natural language and in this report we investigate its potential for time series. We first lift PT into the Spatial-Temporal Probabilistic Transformer (ST-PT) to repair PT's missing channel axis and weak per-step semantics, and adopt ST-PT as a shared cornerstone backbone. We then identify three distinct properties that PT/ST-PT offers as a factor-graph model and derive three Research Questions, one per property, that probe how each property can be exploited in time series: RQ1. The graph topology and potentials are direct programmable primitives. Can this be used to inject symbolic time-series priors into ST-PT through structural graph modifications, especially under data scarcity and noise? RQ2. The CRF's factor matrices are the operator's potentials. Can an external condition program these factor matrices on a per-sample basis, so that conditional generation becomes structural rather than feature-level modulation of a fixed one? RQ3. Each MFVI iteration is a Bayesian posterior update on the factor graph. Can this turn the latent transition of latent-space AutoRegressive (AR) forecasting from an opaque MLP into a principled posterior update, and can a CRF teacher distill its latents into the AR student to counter cumulative error? We give one empirical study per question. Together, these three studies position ST-PT as a programmable framework for time-series modeling.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 49 references that are in the map. First posted on arXiv 29 Apr 2026.

Builds on

25 of 49 references.

References · 25

  1. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  2. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  3. 2022Human Motion Diffusion Modelcited by 1,080
  4. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,058
  5. 2016HyperNetworkscited by 1,027
  6. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,013
  7. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 931
  8. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
  9. 2023TSMixer: An All-MLP Architecture for Time Series Forecastingcited by 240
  10. 2023TimeGPT-1cited by 158
  11. 2022ETSformer: Exponential Smoothing Transformers for Time-series Forecastingcited by 148
  12. 2024TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysiscited by 116
13 more
  1. 2022Physics-Informed Machine Learning: A Survey on Problems, Methods and Applicationscited by 92
  2. 2023Vector Quantized Time Series Generation with a Bidirectional Prior Modelcited by 50
  3. 2017Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecastingcited by 41
  4. 2024Time Weaver: A Conditional Time Series Generation Modelcited by 37
  5. 2024In-Context Fine-Tuning for Time-Series Foundation Modelscited by 35
  6. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 31
  7. 2022TTS-CGAN: A Transformer Time-Series Conditional GAN for Biosignal Data Augmentationcited by 21
  8. 2025T2S: High-resolution Time Series Generation with Text-to-Series Diffusion Modelscited by 20
  9. 2025BRIDGE: Bootstrapping Text to Control Time-Series Generation via Multi-Agent Iterative Optimization and Diffusion Modelingcited by 19
  10. 2020Few-shot Learning for Time-series Forecastingcited by 16
  11. 2025Why Do Transformers Fail to Forecast Time Series In-Context?cited by 5
  12. 2026ConTSG-Bench: A Unified Benchmark for Conditional Time Series Generationcited by 4
  13. 2024Comparing Prior and Learned Time Representations in Transformer Models of Timeseriescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Fighter: Unveiling the Graph Convolutional Nature of Transformers in Time Series Modeling
  2. 2026Superposition Is Not Necessary: A Mechanistic Interpretability Analysis of Transformer Representations for Time Series Forecasting
  3. 2024A Survey of Transformer Enabled Time Series Synthesis
  4. 2024A Temporal Kolmogorov-Arnold Transformer for Time Series Forecasting
  5. 2024TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Model
  6. 2024SST: Multi-Scale Hybrid Mamba-Transformer Experts for Time Series Forecasting
  7. 2026Composing Non-Conjugate Factor Graphs with Closed-Form Variational Inference
  8. 2025S2TX: Cross-Attention Multi-Scale State-Space Transformer for Time Series Forecasting