Hierarchical Multimodal LLMs with Semantic Space Alignment for Enhanced Time Series Classification
On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection
Abstract (arXiv)
Time series classification plays a fundamental role in a wide range of real-world applications. Recently, large language models (LLMs) have demonstrated strong generalization and reasoning capacities, but directly applying them to time series classification remains non-trivial due to the representation gap between numerical sequences and linguistic semantics. In this paper, we propose HiTime, a hierarchical LLM-based framework for multimodal time series classification that bridges structured temporal representations with semantic reasoning in a generative paradigm. Specifically, we design a hierarchical sequence feature encoding module composed of a data-specific encoder and a task-specific encoder to extract complementary temporal features. To mitigate the embedding gap between time series representations and textual semantics, we further introduce a semantic space alignment module that jointly performs coarse-grained global modeling and fine-grained cross-modal correspondence. Building upon the above representations, we employ a parameter-efficient supervised fine-tuning strategy to activate the generative classification capability of the algined LLMs, thereby transforming conventional discriminative time series classification into a generative task. Extensive experiments on multiple benchmarks demonstrate that the proposed framework consistently outperforms state-of-the-art baselines. The code is publicly available at https://github.com/Xiaoyu-Tao/HiTime.
Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 79 references that are in the map. First posted on arXiv 24 Oct 2024.
Builds on
36 of 79 references.
References · 36
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
- 2020Reformer: The Efficient Transformercited by 1,658
- 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
- 2024Chronos: Learning the Language of Time Seriescited by 712
- 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 615
- 2018Deep learning for time series classification: a reviewcited by 575
- 2020A Transformer-based Framework for Multivariate Time Series Representation Learningcited by 449
- 2019InceptionTime: Finding AlexNet for Time Series Classificationcited by 386
- 2018The UEA multivariate time series classification archive, 2018cited by 330
- 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
24 more
- 2019Unsupervised Scalable Representation Learning for Multivariate Time Seriescited by 267
- 2024Foundation Models for Time Series Analysis: A Tutorial and Surveycited by 234
- 2020MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classificationcited by 186
- 2023TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Seriescited by 165
- 2020Benchmarking Multivariate Time Series Classification Algorithmscited by 145
- 2023Spatio-Temporal Graph Neural Networks for Predictive Learning in Urban Computing: A Surveycited by 140
- 2023UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecastingcited by 139
- 2016Multi-Scale Convolutional Neural Networks for Time Series Classificationcited by 99
- 2013A Time Series Forest for Classification and Feature Extractioncited by 69
- 2023Deep Learning for Time Series Classification and Extrinsic Regression: A Current Surveycited by 68
- 2024Empowering Time Series Analysis with Large Language Models: A Surveycited by 60
- 2024aeon: a Python toolkit for learning from time seriescited by 46
- 2018A review on distance based time series classificationcited by 40
- 2014Highly comparative feature-based time-series classificationcited by 39
- 2017Learning Compact Recurrent Neural Networks with Block-Term Tensor Decompositioncited by 33
- 2019Semi-Supervised Hierarchical Recurrent Graph Neural Network for City-Wide Parking Availability Predictioncited by 30
- 2024Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Modelscited by 23
- 2023FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classificationcited by 21
- 2024Exploring Large-Scale Language Models to Evaluate EEG-Based Multimodal Data for Mental Healthcited by 13
- 2024Cross-Domain Pre-training with Language Models for Transferable Time Series Representationscited by 12
- 2024Graph Spatiotemporal Process for Multivariate Time Series Anomaly Detection with Missing Valuescited by 10
- 2024Advancing Time Series Classification with Multimodal Language Modelingcited by 10
- 2023ShapeWordNet: An Interpretable Shapelet Neural Network for Physiological Signal Classificationcited by 7
- 2017A Shapelet Transform for Multivariate Time Series Classificationcited by 3
Built on by
Citing papers per year: 2025: 5; 2026: 2.
7 papers.
- 2025A Survey of Reasoning and Agentic Systems in Time Series with Large Language Modelscited by 21
- 2025Can Slow-thinking LLMs Reason Over Time? Empirical Studies in Time Series Forecastingcited by 19
- 2025Towards Cross-Modality Modeling for Time Series Analytics: A Survey in the LLM Eracited by 11
- 2026Time Series Reasoning via Process-Verifiable Thinking Data Synthesis and Scheduling for Tailored LLM Reasoningcited by 3
- 2026Towards Multimodal Time Series Anomaly Detection with Semantic Alignment and Condensed Interactioncited by 3
- 2025Large Language models for Time Series Analysis: Techniques, Applications, and Challengescited by 2
- 2025From Time Series Analysis to Question Answering: A Survey in the LLM Eracited by 0
- 2026Towards Multimodal Time Series Anomaly Detection with Semantic Alignment and Condensed Interaction23 Mar 2026
- 2026Time Series Reasoning via Process-Verifiable Thinking Data Synthesis and Scheduling for Tailored LLM Reasoning8 Feb 2026
- 2025A Survey of Reasoning and Agentic Systems in Time Series with Large Language Models15 Sep 2025
- 2025From Time Series Analysis to Question Answering: A Survey in the LLM Era13 Jun 2025
- 2025Can Slow-thinking LLMs Reason Over Time? Empirical Studies in Time Series Forecasting30 May 2025
- 2025Large Language models for Time Series Analysis: Techniques, Applications, and Challenges21 May 2025
- 2025Towards Cross-Modality Modeling for Time Series Analytics: A Survey in the LLM Era5 May 2025
Similar, not cited
Close in wording, no citation either way
- 2026InstructTime++: Time Series Classification with Multimodal Language Modeling via Implicit Feature Enhancement
- 2024TableTime: Reformulating Time Series Classification as Training-Free Table Understanding with Large Language Models
- 2024Towards Time Series Reasoning with LLMs
- 2025TimeSense:Making Large Language Models Proficient in Time-Series Analysis
- 2024ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoning
- 2025Adapting LLMs to Time Series Forecasting via Temporal Heterogeneity Modeling and Representation Alignment
- 2024NeST: Neighborhood-aware semantic alignment and temporal modulation for LLM based time series forecasting
- 2025Adapting Large Language Models for Time Series Modeling via a Novel Parameter-efficient Adaptation Method