mlsanityData to 14 Oct 2026

Using Pre-trained LLMs for Multivariate Time Series Forecasting

arXiv 2501.06386 ↗First posted 10 Jan 2025cs.LG cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Pre-trained Large Language Models (LLMs) encapsulate large amounts of knowledge and take enormous amounts of compute to train. We make use of this resource, together with the observation that LLMs are able to transfer knowledge and performance from one domain or even modality to another seemingly-unrelated area, to help with multivariate demand time series forecasting. Attention in transformer-based methods requires something worth attending to -- more than just samples of a time-series. We explore different methods to map multivariate input time series into the LLM token embedding space. In particular, our novel multivariate patching strategy to embed time series features into decoder-only pre-trained Transformers produces results competitive with state-of-the-art time series forecasting models. We also use recently-developed weight-based diagnostics to validate our findings.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 36 of its 42 references that are in the map. First posted on arXiv 10 Jan 2025.

Builds on

36 of 42 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  11. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  12. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
24 more
  1. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
  2. 2020Reformer: The Efficient Transformercited by 1,668
  3. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  4. 2024Chronos: Learning the Language of Time Seriescited by 759
  5. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  6. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 632
  7. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  8. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  9. 2022Transformers in Time Series: A Surveycited by 497
  10. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  11. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  12. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  13. 2017A Multi-Horizon Quantile Recurrent Forecastercited by 211
  14. 2024Are Language Models Actually Useful for Time Series Forecasting?cited by 168
  15. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  16. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  17. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 115
  18. 2020The Early Phase of Neural Network Trainingcited by 115
  19. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 105
  20. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  21. 2022A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translationcited by 80
  22. 2019Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networkscited by 44
  23. 2020MQTransformer: Multi-Horizon Forecasts with Context Dependent and Feedback-Aware Attentioncited by 29
  24. 2022Evaluating natural language processing models with generalization metrics that do not need access to any training or testing datacited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibilitycited by 9
  2. 2025Large Language models for Time Series Analysis: Techniques, Applications, and Challengescited by 3
  3. 2026Rethinking the Role of LLMs in Time Series Forecastingcited by 3
  4. 2025Multivariate Wireless Link Quality Prediction Based on Pre-trained Large Language Modelscited by 0
  5. 2026Forecasting Commencing Enrolments Under Data Sparsity: A Zero-Shot Time Series Foundation Models Framework for Higher Education Planningcited by 0
  6. 2026LLM Pretraining Shapes a Generalizable Manifold: Insights into Cross-Modal Transfer to Time Seriescited by 0