# When Does Text Inform? Benchmarking Information-Theoretic Metrics for Multimodal Time-Series Forecasting

arXiv: https://arxiv.org/abs/2609.11282 · HTML: https://mlsanity.com/paper/2609.11282/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 10 Sep 2026.

## Builds on (most cited first, 18 of 18)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [A Theory of Usable Information Under Computational Constraints](https://mlsanity.com/paper/2002.10689/) (2020)
- [Context is Key: A Benchmark for Forecasting with Essential Textual Information](https://mlsanity.com/paper/2410.18959/) (2024)
- [Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework](https://mlsanity.com/paper/2302.12247/) (2023)
- [Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysis](https://mlsanity.com/paper/2406.08627/) (2024)
- [A Comprehensive Survey of Deep Learning for Time Series Forecasting: Architectural Diversity and Open Challenges](https://mlsanity.com/paper/2411.05793/) (2024)
- [InfoNCE: Identifying the Gap Between Theory and Practice](https://mlsanity.com/paper/2407.00143/) (2024)
- [Beyond Normal: On the Evaluation of Mutual Information Estimators](https://mlsanity.com/paper/2306.11078/) (2023)
- [Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications](https://mlsanity.com/paper/2306.04539/) (2023)
- [CLLMate: A Multimodal Benchmark for Weather and Climate Events Forecasting](https://mlsanity.com/paper/2409.19058/) (2024)
- [RAGEN-2: Reasoning Collapse in Agentic RL](https://mlsanity.com/paper/2604.06268/) (2026)
- [MedTsLLM: Leveraging LLMs for Multimodal Medical Time Series Analysis](https://mlsanity.com/paper/2408.07773/) (2024)
- [Balancing Multimodal Training Through Game-Theoretic Regularization](https://mlsanity.com/paper/2411.07335/) (2024)
- [Demystifying Fixed k-Nearest Neighbor Information Estimators](https://mlsanity.com/paper/1604.03006/) (2016)
- [When Does Multimodality Lead to Better Time Series Forecasting?](https://mlsanity.com/paper/2506.21611/) (2025)
- [FinTexTS: Financial Text-Paired Time-Series Dataset via Semantic-Based and Multi-Level Pairing](https://mlsanity.com/paper/2603.02702/) (2026)

## Abstract (arXiv)

Multimodal forecasting models that combine time series with text annotations promise richer prediction through textual context, but how do we know whether a text annotation meaningfully contributes to the forecasters prediction? This is an information-theoretic question, but to evaluate whether information-theoretic metrics can reliably measure the predictive value an annotation provides, a ground truth benchmark is needed, and none currently exist. We create a synthetic time series signal with annotations in three categories: semantically correct, incorrect, and irrelevant. Because the data generation process is fully controlled, ground-truth information content is known exactly, enabling principled evaluation of six complementary mutual information estimators (KSG, MINE, InfoNCE, CCA, PID and V-information). We show that all six estimators identify correct annotations as most informative, and are able to audit the quality of mixed text corpora, choosing the annotations that result in the best downstream forecasting results without the need for model training. Our benchmark identifies limitations of each estimator, and these are validated on seven real-world datasets, which show how estimator performance differs on weak signals. Finally, we establish practical rules for implementing these metrics for annotation auditing and fusion selection.