mlsanityData to 13 Oct 2026

Towards Long-Context Time Series Foundation Models

arXiv 2409.13530 ↗First posted 20 Sep 2024cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Time series foundation models have shown impressive performance on a variety of tasks, across a wide range of domains, even in zero-shot settings. However, most of these models are designed to handle short univariate time series as an input. This limits their practical use, especially in domains such as healthcare with copious amounts of long and multivariate data with strong temporal and intra-variate dependencies. Our study bridges this gap by cataloging and systematically comparing various context expansion techniques from both language and time series domains, and introducing a novel compressive memory mechanism to allow encoder-only TSFMs to effectively model intra-variate dependencies. We demonstrate the benefits of our approach by imbuing MOMENT, a recent family of multi-task time series foundation models, with the multivariate context.

Key facts. Cited by 5 papers in the map (data through 13 Oct 2026). Builds on 17 of its 22 references that are in the map. First posted on arXiv 20 Sep 2024.

Builds on

17 of 22 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  5. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,016
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,994
  7. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 918
  8. 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 751
  9. 2024Chronos: Learning the Language of Time Seriescited by 737
  10. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 693
  11. 2023A decoder-only foundation model for time-series forecastingcited by 674
  12. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 602
5 more
  1. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 468
  2. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 205
  3. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 197
  4. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 106
  5. 2023Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecastingcited by 105

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Chronos-2: From Univariate to Universal Forecastingcited by 177
  2. 2025Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Modelscited by 15
  3. 2026TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scalecited by 6
  4. 2025Investigating Compositional Reasoning in Time Series Foundation Modelscited by 3
  5. 2026MICA: Multivariate Infini Compressive Attention for Time Series Forecastingcited by 0