# How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlook

arXiv: https://arxiv.org/abs/2503.11835 · HTML: https://mlsanity.com/paper/2503.11835/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 99 of its 217 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Builds on (most cited first, 30 of 99)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019](https://mlsanity.com/paper/1911.13288/) (2019)
- [GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation](https://mlsanity.com/paper/2410.10393/) (2024)
- [TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series](https://mlsanity.com/paper/2308.08241/) (2023)
- [Deep Multi-View Spatial-Temporal Network for Taxi Demand Prediction](https://mlsanity.com/paper/1802.08714/) (2018)
- [A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist](https://mlsanity.com/paper/2402.18485/) (2024)
- [LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters](https://mlsanity.com/paper/2308.08469/) (2023)
- [Large Language Models are Few-Shot Health Learners](https://mlsanity.com/paper/2305.15525/) (2023)
- [A Review on Deep Learning Techniques for Video Prediction](https://mlsanity.com/paper/2004.05214/) (2020)
- [Large Language Models for Time Series: A Survey](https://mlsanity.com/paper/2402.01801/) (2024)
- [ForecastPFN: Synthetically-Trained Zero-Shot Forecasting](https://mlsanity.com/paper/2311.01933/) (2023)
- [ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoning](https://mlsanity.com/paper/2412.03104/) (2024)
- [ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data](https://mlsanity.com/paper/2412.11376/) (2024)
- [From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection](https://mlsanity.com/paper/2409.17515/) (2024)
- [Where Would I Go Next? Large Language Models as Human Mobility Predictors](https://mlsanity.com/paper/2308.15197/) (2023)
- [Voice2Series: Reprogramming Acoustic Models for Time Series Classification](https://mlsanity.com/paper/2106.09296/) (2021)

## Built on by (most cited first, 20 of 20)

- [TimeMaster: Training Time-Series Multimodal LLMs to Reason via Reinforcement Learning](https://mlsanity.com/paper/2506.13705/) (2025)
- [VisionTS++: Cross-Modal Time Series Foundation Model with Continual Pre-trained Vision Backbones](https://mlsanity.com/paper/2508.04379/) (2025)
- [AlphaCast: A Human Wisdom-LLM Intelligence Co-Reasoning Framework for Interactive Time Series Forecasting](https://mlsanity.com/paper/2511.08947/) (2025)
- [TSAQA: Time Series Analysis Question And Answering Benchmark](https://mlsanity.com/paper/2601.23204/) (2026)
- [CaTS-Bench: Can Language Models Describe Time Series?](https://mlsanity.com/paper/2509.20823/) (2025)
- [Time-RA: Towards Time Series Reasoning for Anomaly Diagnosis with LLM Feedback](https://mlsanity.com/paper/2507.15066/) (2025)
- [Who's Your Judge? On the Detectability of LLM-Generated Judgments](https://mlsanity.com/paper/2509.25154/) (2025)
- [From Images to Signals: Are Large Vision Models Useful for Time Series Analysis?](https://mlsanity.com/paper/2505.24030/) (2025)
- [TimeRecipe: A Time-Series Forecasting Recipe via Benchmarking Module Level Effectiveness](https://mlsanity.com/paper/2506.06482/) (2025)
- [Spectral Text Fusion: A Frequency-Aware Approach to Multimodal Time-Series Forecasting](https://mlsanity.com/paper/2602.01588/) (2026)
- [SenTSR-Bench: Thinking with Injected Knowledge for Time-Series Reasoning](https://mlsanity.com/paper/2602.19455/) (2026)
- [Rethinking Post-Training Recipes for Multimodal Time-Series Forecasting](https://mlsanity.com/paper/2605.29401/) (2026)
- [ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction](https://mlsanity.com/paper/2503.20978/) (2025)
- [Causal Graph Fuzzy LLMs: A First Introduction and Applications in Time Series Forecasting](https://mlsanity.com/paper/2507.17016/) (2025)
- [PA-RNet: Perturbation-Aware Residual Network for Robust Multimodal Time Series Forecasting](https://mlsanity.com/paper/2508.04750/) (2025)
- [Multimodal Collaborative Debate for Zero-Shot Time Series Reasoning](https://mlsanity.com/paper/2601.19151/) (2026)
- [A Cost-Effective Multimodal LLM Reasoning Framework for Question Answering over Irregular Clinical Time Series](https://mlsanity.com/paper/2607.25947/) (2026)
- [Transforming Decoder-Only Transformers for Accurate WiFi-Telemetry Based Indoor Localization](https://mlsanity.com/paper/2505.15835/) (2025)
- [Indoor Localization using Compact, Telemetry-Agnostic, Transfer-Learning Enabled Decoder-Only Transformer](https://mlsanity.com/paper/2510.11926/) (2025)
- [PAMF: Prior-Aware Multimodal Fusion for Incomplete Time Series Data](https://mlsanity.com/paper/2606.06328/) (2026)

## Abstract (arXiv)

Time series analysis (TSA) is a longstanding research topic in the data mining community and has wide real-world significance. Compared to "richer" modalities such as language and vision, which have recently experienced explosive development and are densely connected, the time-series modality remains relatively underexplored and isolated. We notice that many recent TSA works have formed a new research field, i.e., Multiple Modalities for TSA (MM4TSA). In general, these MM4TSA works follow a common motivation: how TSA can benefit from multiple modalities. This survey is the first to offer a comprehensive review and a detailed outlook for this emerging field. Specifically, we systematically discuss three benefits: (1) reusing foundation models of other modalities for efficient TSA, (2) multimodal extension for enhanced TSA, and (3) cross-modality interaction for advanced TSA. We further group the works by the introduced modality type, including text, images, audio, tables, and others, within each perspective. Finally, we identify the gaps with future opportunities, including the reused modalities selections, heterogeneous modality combinations, and unseen tasks generalizations, corresponding to the three benefits. We release an up-to-date GitHub repository that includes key papers and resources.