# Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmark

arXiv: https://arxiv.org/abs/2404.16563 · HTML: https://mlsanity.com/paper/2404.16563/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 25 Apr 2024.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [Investigating the Limitations of Transformers with Simple Arithmetic Tasks](https://mlsanity.com/paper/2102.13019/) (2021)
- [How well do Large Language Models perform in Arithmetic tasks?](https://mlsanity.com/paper/2304.02015/) (2023)
- [Large Language Models for Time Series: A Survey](https://mlsanity.com/paper/2402.01801/) (2024)
- [Where Would I Go Next? Large Language Models as Human Mobility Predictors](https://mlsanity.com/paper/2308.15197/) (2023)
- [Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks](https://mlsanity.com/paper/2305.14201/) (2023)
- [Empowering Time Series Analysis with Large Language Models: A Survey](https://mlsanity.com/paper/2402.03182/) (2024)
- [EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model](https://mlsanity.com/paper/2401.10278/) (2024)
- [Foundation Models for Weather and Climate Data Understanding: A Comprehensive Survey](https://mlsanity.com/paper/2312.03014/) (2023)
- [Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?](https://mlsanity.com/paper/2301.09017/) (2023)

## Built on by (most cited first, 30 of 38)

- [ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoning](https://mlsanity.com/paper/2412.03104/) (2024)
- [ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data](https://mlsanity.com/paper/2412.11376/) (2024)
- [Time-MQA: Time Series Multi-Task Question Answering with Context Enhancement](https://mlsanity.com/paper/2503.01875/) (2025)
- [MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answering](https://mlsanity.com/paper/2503.16858/) (2025)
- [A Survey of Reasoning and Agentic Systems in Time Series with Large Language Models](https://mlsanity.com/paper/2509.11575/) (2025)
- [How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlook](https://mlsanity.com/paper/2503.11835/) (2025)
- [Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Models](https://mlsanity.com/paper/2503.11411/) (2025)
- [BEDTime: A Unified Benchmark for Automatically Describing Time Series](https://mlsanity.com/paper/2509.05215/) (2025)
- [TS-Reasoner: Aligning Time Series Foundation Models with LLM Reasoning](https://mlsanity.com/paper/2510.03519/) (2025)
- [CaTS-Bench: Can Language Models Describe Time Series?](https://mlsanity.com/paper/2509.20823/) (2025)
- [TS-Agent: Understanding and Reasoning Over Raw Time Series via Iterative Insight Gathering](https://mlsanity.com/paper/2510.07432/) (2025)
- [CARROT: A Learned Cost-Constrained Retrieval Optimization System for RAG](https://mlsanity.com/paper/2411.00744/) (2024)
- [XForecast: Evaluating Natural Language Explanations for Time Series Forecasting](https://mlsanity.com/paper/2410.14180/) (2024)
- [Language Modeling for the Future of Finance: A Survey into Metrics, Tasks, and Data Opportunities](https://mlsanity.com/paper/2504.07274/) (2025)
- [Towards Interpretable Time Series Foundation Models](https://mlsanity.com/paper/2507.07439/) (2025)
- [ReTabAD: A Benchmark for Restoring Semantic Context in Tabular Anomaly Detection](https://mlsanity.com/paper/2510.02060/) (2025)
- [LTSM-Bundle: A Toolbox and Benchmark on Large Language Models for Time Series Forecasting](https://mlsanity.com/paper/2406.14045/) (2024)
- [MONAQ: Multi-Objective Neural Architecture Querying for Time-Series Analysis on Resource-Constrained Devices](https://mlsanity.com/paper/2505.10607/) (2025)
- [TSVer: A Benchmark for Fact Verification Against Time-Series Evidence](https://mlsanity.com/paper/2511.01101/) (2025)
- [LETS-C: Leveraging Text Embedding for Time Series Classification](https://mlsanity.com/paper/2407.06533/) (2024)
- [ChronoSteer: Bridging Large Language Model and Time Series Foundation Model via Synthetic Cross-Modal Alignment Dataset](https://mlsanity.com/paper/2505.10083/) (2025)
- [AI Analyst: Framework and Comprehensive Evaluation of Large Language Models for Financial Time Series Report Generation](https://mlsanity.com/paper/2507.00718/) (2025)
- [The Forecast Critic: Leveraging Large Language Models for Poor Forecast Identification](https://mlsanity.com/paper/2512.12059/) (2025)
- [Prompting Underestimates LLM Capability for Time Series Classification](https://mlsanity.com/paper/2601.03464/) (2026)
- [DeepArrhythmia: Segment-Contextualized ECG Arrhythmia Classification via Selective Evidence Acquisition](https://mlsanity.com/paper/2605.16441/) (2026)
- [Peak-Detector: Explainable Peak Detection via Instruction-Tuned Large Language Models in Physiological Sign](https://mlsanity.com/paper/2605.16452/) (2026)
- [TS-Skill: A Benchmark for Evaluating Analytical Skills in Time-Series Question Answering](https://mlsanity.com/paper/2605.24703/) (2026)
- [Can LLM Coding Agents Reason About Time Series?](https://mlsanity.com/paper/2606.16545/) (2026)
- [TraceBench: Controlled Evaluation of LLM Agents for Time-Series Root-Cause Attribution](https://mlsanity.com/paper/2608.27182/) (2026)
- [PB-IAD: Utilizing multimodal foundation models for semantic industrial anomaly detection in dynamic manufacturing environments](https://mlsanity.com/paper/2508.14504/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) offer the potential for automatic time series analysis and reporting, which is a critical task across many domains, spanning healthcare, finance, climate, energy, and many more. In this paper, we propose a framework for rigorously evaluating the capabilities of LLMs on time series understanding, encompassing both univariate and multivariate forms. We introduce a comprehensive taxonomy of time series features, a critical framework that delineates various characteristics inherent in time series data. Leveraging this taxonomy, we have systematically designed and synthesized a diverse dataset of time series, embodying the different outlined features, each accompanied by textual descriptions. This dataset acts as a solid foundation for assessing the proficiency of LLMs in comprehending time series. Our experiments shed light on the strengths and limitations of state-of-the-art LLMs in time series understanding, revealing which features these models readily comprehend effectively and where they falter. In addition, we uncover the sensitivity of LLMs to factors including the formatting of the data, the position of points queried within a series and the overall time series length.