# Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data

arXiv: https://arxiv.org/abs/2411.06735 · HTML: https://mlsanity.com/paper/2411.06735/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2024.

## Built on by (most cited first, 22 of 22)

- [Language in the Flow of Time: Time-Series-Paired Texts Weaved into a Unified Temporal Narrative](https://mlsanity.com/paper/2502.08942/) (2025)
- [Multi-modal Time Series Analysis: A Tutorial and Survey](https://mlsanity.com/paper/2503.13709/) (2025)
- [How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlook](https://mlsanity.com/paper/2503.11835/) (2025)
- [Time-IMM: A Dataset and Benchmark for Irregular Multimodal Multivariate Time Series](https://mlsanity.com/paper/2506.10412/) (2025)
- [TimeMaster: Training Time-Series Multimodal LLMs to Reason via Reinforcement Learning](https://mlsanity.com/paper/2506.13705/) (2025)
- [When Does Multimodality Lead to Better Time Series Forecasting?](https://mlsanity.com/paper/2506.21611/) (2025)
- [Diffusion Models for Time Series Forecasting: A Survey](https://mlsanity.com/paper/2507.14507/) (2025)
- [CaTS-Bench: Can Language Models Describe Time Series?](https://mlsanity.com/paper/2509.20823/) (2025)
- [Text Reinforcement for Multimodal Time Series Forecasting](https://mlsanity.com/paper/2509.00687/) (2025)
- [ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Response](https://mlsanity.com/paper/2604.21199/) (2026)
- [AXIS: Explainable Time Series Anomaly Detection with Large Language Models](https://mlsanity.com/paper/2509.24378/) (2025)
- [TsLLM: Augmenting LLMs for General Time Series Understanding and Prediction](https://mlsanity.com/paper/2510.01111/) (2025)
- [Bridging the Last Mile of Time Series Forecasting with LLM Agents](https://mlsanity.com/paper/2606.02497/) (2026)
- [Multimodal Language Models with Modality-Specific Experts for Financial Forecasting from Interleaved Sequences of Text and Time Series](https://mlsanity.com/paper/2509.19628/) (2025)
- [Dr-CiK: A Benchmark testing Deep Research for Context-Aided Forecasting](https://mlsanity.com/paper/2605.27904/) (2026)
- [PA-RNet: Perturbation-Aware Residual Network for Robust Multimodal Time Series Forecasting](https://mlsanity.com/paper/2508.04750/) (2025)
- [LLM-Integrated Bayesian State Space Models for Multimodal Time-Series Forecasting](https://mlsanity.com/paper/2510.20952/) (2025)
- [UniDiff: A Unified Diffusion Framework for Multimodal Time Series Forecasting](https://mlsanity.com/paper/2512.07184/) (2025)
- [Rethinking Multimodal Fusion for Time Series: Text Modalities Need Constrained Fusion](https://mlsanity.com/paper/2603.22372/) (2026)
- [TraceBench: Controlled Evaluation of LLM Agents for Time-Series Root-Cause Attribution](https://mlsanity.com/paper/2608.27182/) (2026)
- [Accurate and Efficient Multi-Channel Time Series Forecasting via Sparse Attention Mechanism](https://mlsanity.com/paper/2603.18712/) (2026)
- [MacroLens: A Multi-Task Benchmark for Contextual Financial Reasoning under Macroeconomic Scenarios](https://mlsanity.com/paper/2606.24950/) (2026)

## Abstract (arXiv)

Current forecasting approaches are largely unimodal and ignore the rich textual data that often accompany the time series due to lack of well-curated multimodal benchmark dataset. In this work, we develop TimeText Corpus (TTC), a carefully curated, time-aligned text and time dataset for multimodal forecasting. Our dataset is composed of sequences of numbers and text aligned to timestamps, and includes data from two different domains: climate science and healthcare. Our data is a significant contribution to the rare selection of available multimodal datasets. We also propose the Hybrid Multi-Modal Forecaster (Hybrid-MMF), a multimodal LLM that jointly forecasts both text and time series data using shared embeddings. However, contrary to our expectations, our Hybrid-MMF model does not outperform existing baselines in our experiments. This negative result highlights the challenges inherent in multimodal forecasting. Our code and data are available at https://github.com/Rose-STL-Lab/Multimodal_ Forecasting.