# Hierarchical Multimodal LLMs with Semantic Space Alignment for Enhanced Time Series Classification

arXiv: https://arxiv.org/abs/2410.18686 · HTML: https://mlsanity.com/paper/2410.18686/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 36 of its 79 references that are in the map.
- First posted on arXiv 24 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 30 of 36)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [InceptionTime: Finding AlexNet for Time Series Classification](https://mlsanity.com/paper/1909.04939/) (2019)
- [The UEA multivariate time series classification archive, 2018](https://mlsanity.com/paper/1811.00075/) (2018)
- [Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs](https://mlsanity.com/paper/2307.03393/) (2023)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Foundation Models for Time Series Analysis: A Tutorial and Survey](https://mlsanity.com/paper/2403.14735/) (2024)
- [MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classification](https://mlsanity.com/paper/2012.08791/) (2020)
- [TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series](https://mlsanity.com/paper/2308.08241/) (2023)
- [Benchmarking Multivariate Time Series Classification Algorithms](https://mlsanity.com/paper/2007.13156/) (2020)
- [UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting](https://mlsanity.com/paper/2310.09751/) (2023)
- [Spatio-Temporal Graph Neural Networks for Predictive Learning in Urban Computing: A Survey](https://mlsanity.com/paper/2303.14483/) (2023)
- [Multi-Scale Convolutional Neural Networks for Time Series Classification](https://mlsanity.com/paper/1603.06995/) (2016)
- [A Time Series Forest for Classification and Feature Extraction](https://mlsanity.com/paper/1302.2277/) (2013)
- [Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey](https://mlsanity.com/paper/2302.02515/) (2023)
- [Empowering Time Series Analysis with Large Language Models: A Survey](https://mlsanity.com/paper/2402.03182/) (2024)
- [aeon: a Python toolkit for learning from time series](https://mlsanity.com/paper/2406.14231/) (2024)
- [Highly comparative feature-based time-series classification](https://mlsanity.com/paper/1401.3531/) (2014)
- [A review on distance based time series classification](https://mlsanity.com/paper/1806.04509/) (2018)
- [Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition](https://mlsanity.com/paper/1712.05134/) (2017)
- [Semi-Supervised Hierarchical Recurrent Graph Neural Network for City-Wide Parking Availability Prediction](https://mlsanity.com/paper/1911.10516/) (2019)
- [Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Models](https://mlsanity.com/paper/2402.01749/) (2024)
- [FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification](https://mlsanity.com/paper/2302.09818/) (2023)

## Built on by (most cited first, 7 of 7)

- [A Survey of Reasoning and Agentic Systems in Time Series with Large Language Models](https://mlsanity.com/paper/2509.11575/) (2025)
- [Can Slow-thinking LLMs Reason Over Time? Empirical Studies in Time Series Forecasting](https://mlsanity.com/paper/2505.24511/) (2025)
- [Towards Cross-Modality Modeling for Time Series Analytics: A Survey in the LLM Era](https://mlsanity.com/paper/2505.02583/) (2025)
- [Large Language models for Time Series Analysis: Techniques, Applications, and Challenges](https://mlsanity.com/paper/2506.11040/) (2025)
- [Time Series Reasoning via Process-Verifiable Thinking Data Synthesis and Scheduling for Tailored LLM Reasoning](https://mlsanity.com/paper/2602.07830/) (2026)
- [Towards Multimodal Time Series Anomaly Detection with Semantic Alignment and Condensed Interaction](https://mlsanity.com/paper/2603.21612/) (2026)
- [From Time Series Analysis to Question Answering: A Survey in the LLM Era](https://mlsanity.com/paper/2506.11512/) (2025)

## Abstract (arXiv)

Time series classification plays a fundamental role in a wide range of real-world applications. Recently, large language models (LLMs) have demonstrated strong generalization and reasoning capacities, but directly applying them to time series classification remains non-trivial due to the representation gap between numerical sequences and linguistic semantics. In this paper, we propose HiTime, a hierarchical LLM-based framework for multimodal time series classification that bridges structured temporal representations with semantic reasoning in a generative paradigm. Specifically, we design a hierarchical sequence feature encoding module composed of a data-specific encoder and a task-specific encoder to extract complementary temporal features. To mitigate the embedding gap between time series representations and textual semantics, we further introduce a semantic space alignment module that jointly performs coarse-grained global modeling and fine-grained cross-modal correspondence. Building upon the above representations, we employ a parameter-efficient supervised fine-tuning strategy to activate the generative classification capability of the algined LLMs, thereby transforming conventional discriminative time series classification into a generative task. Extensive experiments on multiple benchmarks demonstrate that the proposed framework consistently outperforms state-of-the-art baselines. The code is publicly available at https://github.com/Xiaoyu-Tao/HiTime.