mlsanityData to 7 Oct 2026

TimeSeriesExam: A time series understanding exam

arXiv 2410.14752 ↗First posted 18 Oct 2024cs.AI cs.CLcited by 47

Not on the map. Why?

Cited by 21 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) have recently demonstrated a remarkable ability to model time series data. These capabilities can be partly explained if LLMs understand basic time series concepts. However, our knowledge of what these models understand about time series data remains relatively limited. To address this gap, we introduce TimeSeriesExam, a configurable and scalable multiple-choice question exam designed to assess LLMs across five core time series understanding categories: pattern recognition, noise understanding, similarity analysis, anomaly detection, and causality analysis. TimeSeriesExam comprises of over 700 questions, procedurally generated using 104 carefully curated templates and iteratively refined to balance difficulty and their ability to discriminate good from bad models. We test 7 state-of-the-art LLMs on the TimeSeriesExam and provide the first comprehensive evaluation of their time series understanding abilities. Our results suggest that closed-source models such as GPT-4 and Gemini understand simple time series concepts significantly better than their open-source counterparts, while all models struggle with complex concepts such as causality analysis. We believe that the ability to programatically generate questions is fundamental to assessing and improving LLM's ability to understand and reason about time series data.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

22 of 30 references.

References · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  3. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  6. 2024Chronos: Learning the Language of Time Seriescited by 712
  7. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 680
  8. 2023A decoder-only foundation model for time-series forecastingcited by 650
  9. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 615
  10. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
  11. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  12. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 453
10 more
  1. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 198
  2. 2023TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecastingcited by 170
  3. 2023TimeGPT-1cited by 149
  4. 2022CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecastingcited by 143
  5. 2020Benchmarking Deep Learning Interpretability in Time Series Predictionscited by 102
  6. 2024Language Models Still Struggle to Zero-shot Reason about Time Seriescited by 81
  7. 2023ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogramcited by 52
  8. 2024Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generationcited by 25
  9. 2022Unsupervised Model Selection for Time-series Anomaly Detectioncited by 24
  10. 2022py-irt: A Scalable Item Response Theory Library for Pythoncited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 23; 2026: 23.

47 papers.

Sort:

By area:

  1. 2025MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answeringcited by 50
  2. 2025TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loopcited by 39
  3. 2025ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Datasetcited by 35
  4. 2024A Picture is Worth A Thousand Numbers: Enabling LLMs Reason about Time Series via Visualizationcited by 31
  5. 2026TimeART: Towards Agentic Time Series Reasoning via Tool-Augmentationcited by 25
  6. 2025A Survey of Reasoning and Agentic Systems in Time Series with Large Language Modelscited by 21
  7. 2025How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlookcited by 20
  8. 2025Achieving Time Series Reasoning Requires Rethinking Model Design, Tasks Formulation, and Evaluationcited by 15
  9. 2025Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Modelscited by 15
  10. 2025TimeSeriesGym: A Scalable Benchmark for (Time Series) Machine Learning Engineering Agentscited by 14
  11. 2026TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Modelscited by 14
  12. 2025SciTS: Scientific Time Series Understanding and Generation with LLMscited by 13
35 more
  1. 2025BEDTime: A Unified Benchmark for Automatically Describing Time Seriescited by 11
  2. 2025Inferring Events from Time Series using Language Modelscited by 10
  3. 2025TS-Reasoner: Aligning Time Series Foundation Models with LLM Reasoningcited by 10
  4. 2025Towards Foundation Models for Zero-Shot Time Series Anomaly Detection: Leveraging Synthetic Data and Relative Context Discrepancycited by 9
  5. 2026TSAQA: Time Series Analysis Question And Answering Benchmarkcited by 8
  6. 2026HEARTS: Benchmarking LLM Reasoning on Health Time Seriescited by 8
  7. 2025TempoGPT: Enhancing Time Series Reasoning via Quantizing Embeddingcited by 7
  8. 2025TS-Agent: Understanding and Reasoning Over Raw Time Series via Iterative Insight Gatheringcited by 7
  9. 2026STReasoner: Empowering LLMs for Spatio-Temporal Reasoning in Time Series via Spatial-Aware Reinforcement Learningcited by 7
  10. 2026ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Responsecited by 5
  11. 2025AXIS: Explainable Time Series Anomaly Detection with Large Language Modelscited by 4
  12. 2026CLIR-Bench: Benchmarking Multimodal Question Answering over Irregular Clinical Time Seriescited by 4
  13. 2025Investigating Compositional Reasoning in Time Series Foundation Modelscited by 3
  14. 2026Time Series Reasoning via Process-Verifiable Thinking Data Synthesis and Scheduling for Tailored LLM Reasoningcited by 3
  15. 2026LLaTiSA: Towards Difficulty-Stratified Time Series Reasoning from Visual Perception to Semanticscited by 3
  16. 2025Refining Time Series Anomaly Detectors using Large Language Modelscited by 2
  17. 2025MONAQ: Multi-Objective Neural Architecture Querying for Time-Series Analysis on Resource-Constrained Devicescited by 2
  18. 2025Rating Quality of Diverse Time Series Data by Meta-learning from LLM Judgmentcited by 1
  19. 2025LLM-Integrated Bayesian State Space Models for Multimodal Time-Series Forecastingcited by 1
  20. 2026TS-Haystack: A Multi-Task Retrieval Benchmark for Long-Context Time-Series Reasoningcited by 1
  21. 2026Generating Expressive and Customizable Evals for Timeseries Data Analysis Agents with AgentFuelcited by 1
  22. 2026TSQAgent: Rating Time Series Data Quality via Dedicated Agentic Reasoningcited by 1
  23. 2026LLM Agents for Time-Series: A Surveycited by 1
  24. 2025ChronoSteer: Bridging Large Language Model and Time Series Foundation Model via Synthetic Cross-Modal Alignment Datasetcited by 0
  25. 2025Can VLM Pseudo-Labels Train a Time-Series QA Model That Outperforms the VLM?cited by 0
  26. 2026KairosVL: Orchestrating Time Series and Semantics for Unified Reasoningcited by 0
  27. 2026TimeRLM: Recursive Language Models Enable Precise Anomaly Localization in Long-Context Time-Seriescited by 0
  28. 2026TimeSage-EV: A Live Benchmark for Agentic Time Series Analysis in Evolving Environmentscited by 0
  29. 2026WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Datacited by 0
  30. 2026HealthLoopQA: A Context-Aware Question Answering Benchmark for Interpreting Wearable Monitoring Data in Diabetes Carecited by 0
  31. 2026TimeThink: Eliciting Compositional Reasoning in Timeseries Large Language Modelscited by 0
  32. 2026T-SMART: Mechanism-Level Attribution for Tool-Augmented Time-Series Question Answeringcited by 0
  33. 2026TimeInteract: Towards Real-Time Interactive Intelligence for Streaming Time Seriescited by 0
  34. 2026TimeEvo: Failure-Driven Self-Evolution of a Time Series Agentcited by 0
  35. 2026OpenTSLM TeeMoE: A Unified Time-Series Language Model for Forecasting, Contextual Prediction, and Reasoningcited by 0