# Rethinking Multimodal Fusion for Time Series: Text Modalities Need Constrained Fusion

arXiv: https://arxiv.org/abs/2603.22372 · HTML: https://mlsanity.com/paper/2603.22372/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 47 references that are in the map.
- First posted on arXiv 23 Mar 2026.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Long-term Forecasting with TiDE: Time-series Dense Encoder](https://mlsanity.com/paper/2304.08424/) (2023)
- [Language in the Flow of Time: Time-Series-Paired Texts Weaved into a Unified Temporal Narrative](https://mlsanity.com/paper/2502.08942/) (2025)
- [Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data](https://mlsanity.com/paper/2411.06735/) (2024)
- [Multimodal Conditioned Diffusive Time Series Forecasting](https://mlsanity.com/paper/2504.19669/) (2025)
- [T3Time: Tri-Modal Time Series Forecasting via Adaptive Multi-Head Alignment and Residual Fusion](https://mlsanity.com/paper/2508.04251/) (2025)
- [BALM-TSF: Balanced Multimodal Alignment for LLM-Based Time Series Forecasting](https://mlsanity.com/paper/2509.00622/) (2025)
- [Spectral Text Fusion: A Frequency-Aware Approach to Multimodal Time-Series Forecasting](https://mlsanity.com/paper/2602.01588/) (2026)
- [Context-Aware Probabilistic Modeling with LLM for Multimodal Time Series Forecasting](https://mlsanity.com/paper/2505.10774/) (2025)
- [TiMi: Empower Time Series Transformers with Multimodal Mixture of Experts](https://mlsanity.com/paper/2602.21693/) (2026)
- [Context Matters: Leveraging Contextual Features for Time Series Forecasting](https://mlsanity.com/paper/2410.12672/) (2024)
- [UniCast: A Unified Framework for Instance-Conditioned Multimodal Time-Series Forecasting](https://mlsanity.com/paper/2508.11954/) (2025)
- [PA-RNet: Perturbation-Aware Residual Network for Robust Multimodal Time Series Forecasting](https://mlsanity.com/paper/2508.04750/) (2025)

## Built on by (most cited first, 1 of 1)

- [WorldTS: World Modeling for Multimodal Covariate-aware Time Series Forecasting](https://mlsanity.com/paper/2609.31162/) (2026)

## Abstract (arXiv)

Recent advances in multimodal learning have motivated the integration of auxiliary modalities such as text or vision into time series (TS) forecasting. However, most existing methods provide limited gains, often improving performance only in specific datasets or relying on architecture-specific designs that limit generalization. In this paper, we show that multimodal models with naive fusion strategies (e.g., simple addition or concatenation) often underperform unimodal TS models, which we attribute to the uncontrolled integration of auxiliary modalities which may introduce irrelevant information. Motivated by this observation, we explore various constrained fusion methods designed to control such integration and find that they consistently outperform naive fusion methods. Furthermore, we propose Controlled Fusion Adapter (CFA), a simple plug-in method that enables controlled cross-modal interactions without modifying the TS backbone, integrating only relevant textual information aligned with TS dynamics. CFA employs low rank adapters to filter irrelevant textual information before fusing it into temporal representations. We conduct over 20K experiments across various datasets and TS/text models, demonstrating the effectiveness of the constrained fusion methods. Code is available at: https://github.com/seunghan96/cfa.