mlsanityData to 7 Oct 2026

tsGT: Stochastic Time Series Modeling With Transformer

arXiv 2403.05713 ↗First posted 8 Mar 2024cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Time series methods are of fundamental importance in virtually any field of science that deals with temporally structured data. Recently, there has been a surge of deterministic transformer models with time series-specific architectural biases. In this paper, we go in a different direction by introducing tsGT, a stochastic time series model built on a general-purpose transformer architecture. We focus on using a well-known and theoretically justified rolling window backtesting and evaluation protocol. We show that tsGT outperforms the state-of-the-art models on MAD and RMSE, and surpasses its stochastic peers on QL and CRPS, on four commonly used datasets. We complement these results with a detailed analysis of tsGT's ability to model the data distribution and predict marginal quantile values.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 42 of its 68 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

42 of 68 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2023A Survey of Large Language Modelscited by 2,537
  12. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
30 more
  1. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  2. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  6. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  8. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,004
  9. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  10. 2022A Generalist Agentcited by 802
  11. 2020Efficient Transformers: A Surveycited by 784
  12. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  13. 2019Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecastingcited by 748
  14. 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 744
  15. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  16. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 606
  17. 2022Transformers in Time Series: A Surveycited by 489
  18. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  19. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  20. 2023TSMixer: An All-MLP Architecture for Time Series Forecastingcited by 232
  21. 2021BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG datacited by 194
  22. 2022Formal Mathematics Statement Curriculum Learningcited by 141
  23. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  24. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  25. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  26. 2023Magnushammer: A Transformer-Based Approach to Premise Selectioncited by 50
  27. 2021CRPS Learningcited by 21
  28. 2021Subgoal Search For Complex Reasoning Taskscited by 21
  29. 2021Probabilistic Time Series Forecasting with Implicit Quantile Networkscited by 17
  30. 2022Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Searchcited by 12

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Foundation Model Forecasts: Form and Functioncited by 0