# When Does Multimodality Lead to Better Time Series Forecasting?

arXiv: https://arxiv.org/abs/2506.21611 · HTML: https://mlsanity.com/paper/2506.21611/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 20 Jun 2025.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting](https://mlsanity.com/paper/2310.04948/) (2023)
- [PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecasting](https://mlsanity.com/paper/2210.08964/) (2022)
- [TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series](https://mlsanity.com/paper/2308.08241/) (2023)
- [UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting](https://mlsanity.com/paper/2310.09751/) (2023)
- [Large Language Models for Time Series: A Survey](https://mlsanity.com/paper/2402.01801/) (2024)
- [ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data](https://mlsanity.com/paper/2412.11376/) (2024)
- [Language Models Still Struggle to Zero-shot Reason about Time Series](https://mlsanity.com/paper/2404.11757/) (2024)
- [Context is Key: A Benchmark for Forecasting with Essential Textual Information](https://mlsanity.com/paper/2410.18959/) (2024)
- [AutoTimes: Autoregressive Time Series Forecasters via Large Language Models](https://mlsanity.com/paper/2402.02370/) (2024)
- [LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language](https://mlsanity.com/paper/2405.12856/) (2024)
- [MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answering](https://mlsanity.com/paper/2503.16858/) (2025)
- [Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data](https://mlsanity.com/paper/2411.06735/) (2024)
- [Well Googled is Half Done: Multimodal Forecasting of New Fashion Product Sales with Image-based Google Trends](https://mlsanity.com/paper/2109.09824/) (2021)
- [LLMForecaster: Improving Seasonal Event Forecasts with Unstructured Textual Data](https://mlsanity.com/paper/2412.02525/) (2024)

## Built on by (most cited first, 11 of 11)

- [Chronos-2: From Univariate to Universal Forecasting](https://mlsanity.com/paper/2510.15821/) (2025)
- [A Survey of Reasoning and Agentic Systems in Time Series with Large Language Models](https://mlsanity.com/paper/2509.11575/) (2025)
- [Overcoming the Modality Gap in Context-Aided Forecasting](https://mlsanity.com/paper/2603.12451/) (2026)
- [EventTSF: Event-Aware Non-Stationary Time Series Forecasting](https://mlsanity.com/paper/2508.13434/) (2025)
- [Multi-Modal Time Series Prediction via Mixture of Modulated Experts](https://mlsanity.com/paper/2601.21547/) (2026)
- [Toward Privileged Foundation Models:LUPI for Accelerated and Improved Learning](https://mlsanity.com/paper/2605.07799/) (2026)
- [Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding](https://mlsanity.com/paper/2605.20268/) (2026)
- [Semantics or Structure? Auditing Text Sensitivity in Multimodal Time-Series Forecasting](https://mlsanity.com/paper/2608.22321/) (2026)
- [When Does Text Inform? Benchmarking Information-Theoretic Metrics for Multimodal Time-Series Forecasting](https://mlsanity.com/paper/2609.11282/) (2026)
- [OpenTSLM TeeMoE: A Unified Time-Series Language Model for Forecasting, Contextual Prediction, and Reasoning](https://mlsanity.com/paper/2609.40265/) (2026)
- [Do Time-Series QA Systems Read the Time Series? Evidence Use and Reasoning Reliability](https://mlsanity.com/paper/2610.05686/) (2026)

## Abstract (arXiv)

Recently, there has been growing interest in incorporating textual information into foundation models for time series forecasting. However, it remains unclear whether and under what conditions such multimodal integration consistently yields gains. We systematically investigate these questions across a diverse benchmark of 16 forecasting tasks spanning 7 domains, including health, environment, and economics. We evaluate two popular multimodal forecasting paradigms: aligning-based methods, which align time series and text representations; and prompting-based methods, which directly prompt large language models for forecasting. Our findings reveal that the benefits of multimodality are highly condition-dependent. While we confirm reported gains in some settings, these improvements are not universal across datasets or models. To move beyond empirical observations, we disentangle the effects of model architectural properties and data characteristics, drawing data-agnostic insights that generalize across domains. Our findings highlight that on the modeling side, incorporating text information is most helpful given (1) high-capacity text models, (2) comparatively weaker time series models, and (3) appropriate aligning strategies. On the data side, performance gains are more likely when (4) sufficient training data is available and (5) the text offers complementary predictive signal beyond what is already captured from the time series alone. Our study offers a rigorous, quantitative foundation for understanding when multimodality can be expected to aid forecasting tasks, and reveals that its benefits are neither universal nor always aligned with intuition.