# Understanding Why Large Language Models Can Be Ineffective in Time Series Analysis: The Impact of Modality Alignment

arXiv: https://arxiv.org/abs/2410.12326 · HTML: https://mlsanity.com/paper/2410.12326/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Builds on (most cited first, 19 of 19)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [Time Series Forecasting With Deep Learning: A Survey](https://mlsanity.com/paper/2004.13408/) (2020)
- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [Are Language Models Actually Useful for Time Series Forecasting?](https://mlsanity.com/paper/2406.16964/) (2024)
- [TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2306.09364/) (2023)
- [LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters](https://mlsanity.com/paper/2308.08469/) (2023)
- [CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuning](https://mlsanity.com/paper/2403.07300/) (2024)
- [Revolutionizing Finance with LLMs: An Overview of Applications and Insights](https://mlsanity.com/paper/2401.11641/) (2024)
- [Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning](https://mlsanity.com/paper/2312.11420/) (2023)
- [An Analysis of Linear Time Series Forecasting Models](https://mlsanity.com/paper/2403.14587/) (2024)
- [SWAP: Exploiting Second-Ranked Logits for Adversarial Attacks on Time Series](https://mlsanity.com/paper/2309.02752/) (2023)

## Built on by (most cited first, 7 of 7)

- [MemoTime: Memory-Augmented Temporal Knowledge Graph Enhanced Large Language Model Reasoning](https://mlsanity.com/paper/2510.13614/) (2025)
- [TempoGPT: Enhancing Time Series Reasoning via Quantizing Embedding](https://mlsanity.com/paper/2501.07335/) (2025)
- [Lifting Manifolds to Mitigate Pseudo-Alignment in LLM4TS](https://mlsanity.com/paper/2510.12847/) (2025)
- [LLM Pretraining Shapes a Generalizable Manifold: Insights into Cross-Modal Transfer to Time Series](https://mlsanity.com/paper/2605.20449/) (2026)
- [A Study of Temporal Fusion Strategies for Named Entity Recognition in Historical Texts](https://mlsanity.com/paper/2606.27881/) (2026)
- [SCENARIODIFF: A Scenario-level Guidance Framework for Multimodal Time Series Forecasting--Extended Version](https://mlsanity.com/paper/2608.17164/) (2026)
- [Structured Frequency-Domain Evidence for LLM-Based Time-Series Anomaly Detection](https://mlsanity.com/paper/2608.24113/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated impressive performance in time series analysis and seems to understand the time temporal relationship well than traditional transformer-based approaches. However, since LLMs are not designed for time series tasks, simpler models like linear regressions can often achieve comparable performance with far less complexity. In this study, we perform extensive experiments to assess the effectiveness of applying LLMs to key time series tasks, including forecasting, classification, imputation, and anomaly detection. We compare the performance of LLMs against simpler baseline models, such as single layer linear models and randomly initialized LLMs. Our results reveal that LLMs offer minimal advantages for these core time series tasks and may even distort the temporal structure of the data. In contrast, simpler models consistently outperform LLMs while requiring far fewer parameters. Furthermore, we analyze existing reprogramming techniques and show, through data manifold analysis, that these methods fail to effectively align time series data with language and display "pseudo-alignment" behavior in embedding space. Our findings suggest that the performance of LLM based methods in time series tasks arises from the intrinsic characteristics and structure of time series data, rather than any meaningful alignment with the language model architecture.