mlsanityData to 14 Oct 2026

Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data

arXiv 2411.06735 ↗First posted 11 Nov 2024cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

Current forecasting approaches are largely unimodal and ignore the rich textual data that often accompany the time series due to lack of well-curated multimodal benchmark dataset. In this work, we develop TimeText Corpus (TTC), a carefully curated, time-aligned text and time dataset for multimodal forecasting. Our dataset is composed of sequences of numbers and text aligned to timestamps, and includes data from two different domains: climate science and healthcare. Our data is a significant contribution to the rare selection of available multimodal datasets. We also propose the Hybrid Multi-Modal Forecaster (Hybrid-MMF), a multimodal LLM that jointly forecasts both text and time series data using shared embeddings. However, contrary to our expectations, our Hybrid-MMF model does not outperform existing baselines in our experiments. This negative result highlights the challenges inherent in multimodal forecasting. Our code and data are available at https://github.com/Rose-STL-Lab/Multimodal_ Forecasting.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Nov 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 7.

22 papers.

Sort:

By area:

  1. 2025Language in the Flow of Time: Time-Series-Paired Texts Weaved into a Unified Temporal Narrativecited by 40
  2. 2025Multi-modal Time Series Analysis: A Tutorial and Surveycited by 24
  3. 2025How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlookcited by 20
  4. 2025Time-IMM: A Dataset and Benchmark for Irregular Multimodal Multivariate Time Seriescited by 19
  5. 2025TimeMaster: Training Time-Series Multimodal LLMs to Reason via Reinforcement Learningcited by 18
  6. 2025When Does Multimodality Lead to Better Time Series Forecasting?cited by 11
  7. 2025Diffusion Models for Time Series Forecasting: A Surveycited by 11
  8. 2025CaTS-Bench: Can Language Models Describe Time Series?cited by 10
  9. 2025Text Reinforcement for Multimodal Time Series Forecastingcited by 7
  10. 2026ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Responsecited by 6
  11. 2025AXIS: Explainable Time Series Anomaly Detection with Large Language Modelscited by 4
  12. 2025TsLLM: Augmenting LLMs for General Time Series Understanding and Predictioncited by 4
10 more
  1. 2026Bridging the Last Mile of Time Series Forecasting with LLM Agentscited by 3
  2. 2025Multimodal Language Models with Modality-Specific Experts for Financial Forecasting from Interleaved Sequences of Text and Time Seriescited by 2
  3. 2026Dr-CiK: A Benchmark testing Deep Research for Context-Aided Forecastingcited by 2
  4. 2025PA-RNet: Perturbation-Aware Residual Network for Robust Multimodal Time Series Forecastingcited by 1
  5. 2025LLM-Integrated Bayesian State Space Models for Multimodal Time-Series Forecastingcited by 1
  6. 2025UniDiff: A Unified Diffusion Framework for Multimodal Time Series Forecastingcited by 1
  7. 2026Rethinking Multimodal Fusion for Time Series: Text Modalities Need Constrained Fusioncited by 1
  8. 2026TraceBench: Controlled Evaluation of LLM Agents for Time-Series Root-Cause Attributioncited by 1
  9. 2026Accurate and Efficient Multi-Channel Time Series Forecasting via Sparse Attention Mechanismcited by 0
  10. 2026MacroLens: A Multi-Task Benchmark for Contextual Financial Reasoning under Macroeconomic Scenarioscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025BALM-TSF: Balanced Multimodal Alignment for LLM-Based Time Series Forecasting
  2. 2026TiMi: Empower Time Series Transformers with Multimodal Mixture of Experts
  3. 2025Enhancing Time Series Forecasting via Multi-Level Text Alignment with LLMs
  4. 2024TS-HTFA: Advancing Time Series Forecasting via Hierarchical Text-Free Alignment with Large Language Models
  5. 2024TimeCMA: Towards LLM-Empowered Multivariate Time Series Forecasting via Cross-Modality Alignment
  6. 2025Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectives
  7. 2025Aurora: Towards Universal Generative Multimodal Time Series Forecasting
  8. 2025DualSG: A Dual-Stream Explicit Semantic-Guided Multivariate Time Series Forecasting Framework