# Overcoming the Modality Gap in Context-Aided Forecasting

arXiv: https://arxiv.org/abs/2603.12451 · HTML: https://mlsanity.com/paper/2603.12451/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 12 Mar 2026.

## Builds on (most cited first, 27 of 27)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation](https://mlsanity.com/paper/2410.10393/) (2024)
- [Chronos-2: From Univariate to Universal Forecasting](https://mlsanity.com/paper/2510.15821/) (2025)
- [PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecasting](https://mlsanity.com/paper/2210.08964/) (2022)
- [Are Language Models Actually Useful for Time Series Forecasting?](https://mlsanity.com/paper/2406.16964/) (2024)
- [UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting](https://mlsanity.com/paper/2310.09751/) (2023)
- [TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysis](https://mlsanity.com/paper/2410.16032/) (2024)
- [ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data](https://mlsanity.com/paper/2412.11376/) (2024)
- [From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection](https://mlsanity.com/paper/2409.17515/) (2024)
- [Language Models Still Struggle to Zero-shot Reason about Time Series](https://mlsanity.com/paper/2404.11757/) (2024)
- [Context is Key: A Benchmark for Forecasting with Essential Textual Information](https://mlsanity.com/paper/2410.18959/) (2024)
- [Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysis](https://mlsanity.com/paper/2406.08627/) (2024)
- [AutoTimes: Autoregressive Time Series Forecasters via Large Language Models](https://mlsanity.com/paper/2402.02370/) (2024)
- [Time-VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting](https://mlsanity.com/paper/2502.04395/) (2025)
- [LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language](https://mlsanity.com/paper/2405.12856/) (2024)
- [MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answering](https://mlsanity.com/paper/2503.16858/) (2025)
- [Aurora: Towards Universal Generative Multimodal Time Series Forecasting](https://mlsanity.com/paper/2509.22295/) (2025)
- [Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning](https://mlsanity.com/paper/2402.04852/) (2024)
- [T2S: High-resolution Time Series Generation with Text-to-Series Diffusion Models](https://mlsanity.com/paper/2505.02417/) (2025)
- [FinMultiTime: A Four-Modal Bilingual Dataset for Financial Time-Series Analysis](https://mlsanity.com/paper/2506.05019/) (2025)
- [When Does Multimodality Lead to Better Time Series Forecasting?](https://mlsanity.com/paper/2506.21611/) (2025)
- [Dual-Forecaster: A Multimodal Time Series Model Integrating Descriptive and Predictive Texts](https://mlsanity.com/paper/2505.01135/) (2025)

## Built on by (most cited first, 5 of 5)

- [TimeBraid: Unifying Time Series and Language for Understanding and Forecasting](https://mlsanity.com/paper/2609.29792/) (2026)
- [OpenTSLM TeeMoE: A Unified Time-Series Language Model for Forecasting, Contextual Prediction, and Reasoning](https://mlsanity.com/paper/2609.40265/) (2026)
- [TIMBRE: Teaching Time Series Forecasters to Read, Remember, and Reconcile](https://mlsanity.com/paper/2610.04795/) (2026)
- [Pythia: Toward Foundation World Models for Multimodal Time Series](https://mlsanity.com/paper/2610.05240/) (2026)
- [Omni-Modal LLMs Know More Than They Say](https://mlsanity.com/paper/2610.16135/) (2026)

## Abstract (arXiv)

Context-aided forecasting (CAF) holds promise for integrating domain knowledge and forward-looking information, enabling AI systems to surpass traditional statistical methods. However, recent empirical studies reveal a puzzling gap: multimodal models often fail to outperform their unimodal counterparts. We hypothesize that this underperformance stems from poor context quality in existing datasets, as verification is challenging. To address these limitations, we introduce a semi-synthetic data augmentation method that generates contexts both descriptive of temporal dynamics and verifiably complementary to numerical histories. This approach enables massive-scale dataset creation, resulting in CAF-7M, a corpus of 7 million context-augmented time series windows, including a rigorously verified test set. We demonstrate that semi-synthetic pre-training transfers effectively to real-world evaluation, and show clear evidence of context utilization. Our results suggest that dataset quality, rather than architectural limitations, has been the primary bottleneck in context-aided forecasting.