# Modality-aware Transformer for Financial Time series Forecasting

arXiv: https://arxiv.org/abs/2310.01232 · HTML: https://mlsanity.com/paper/2310.01232/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [Actor-Transformers for Group Activity Recognition](https://mlsanity.com/paper/2003.12737/) (2020)
- [Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline](https://mlsanity.com/paper/1912.02379/) (2019)
- [InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining](https://mlsanity.com/paper/2003.13198/) (2020)
- [Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining](https://mlsanity.com/paper/2107.14572/) (2021)
- [A Deep Adversarial Model for Suffix and Remaining Time Prediction of Event Sequences](https://mlsanity.com/paper/2102.07298/) (2021)
- [Probabilistic Decomposition Transformer for Time Series Forecasting](https://mlsanity.com/paper/2210.17393/) (2022)

## Built on by (most cited first, 7 of 7)

- [Unlocking the Value of Text: Event-Driven Reasoning and Multi-Level Alignment for Time Series Forecasting](https://mlsanity.com/paper/2603.15452/) (2026)
- [EventTSF: Event-Aware Non-Stationary Time Series Forecasting](https://mlsanity.com/paper/2508.13434/) (2025)
- [QiYao-M: Multimodal Time Series Foundation Model with Role-Aware Modeling of Endogenous and Exogenous Modalities](https://mlsanity.com/paper/2609.34842/) (2026)
- [TABL-ABM: A Hybrid Framework for Synthetic LOB Generation](https://mlsanity.com/paper/2510.22685/) (2025)
- [Exposing Vulnerabilities in Explanation for Time Series Classifiers via Dual-Target Attacks](https://mlsanity.com/paper/2602.02763/) (2026)
- [The challenge of generating and evolving real-life like synthetic test data without accessing real-world raw data -- a Systematic Review](https://mlsanity.com/paper/2602.06609/) (2026)
- [Accurate and Efficient Multi-Channel Time Series Forecasting via Sparse Attention Mechanism](https://mlsanity.com/paper/2603.18712/) (2026)

## Abstract (arXiv)

Time series forecasting presents a significant challenge, particularly when its accuracy relies on external data sources rather than solely on historical values. This issue is prevalent in the financial sector, where the future behavior of time series is often intricately linked to information derived from various textual reports and a multitude of economic indicators. In practice, the key challenge lies in constructing a reliable time series forecasting model capable of harnessing data from diverse sources and extracting valuable insights to predict the target time series accurately. In this work, we tackle this challenging problem and introduce a novel multimodal transformer-based model named the textitModality-aware Transformer. Our model excels in exploring the power of both categorical text and numerical timeseries to forecast the target time series effectively while providing insights through its neural attention mechanism. To achieve this, we develop feature-level attention layers that encourage the model to focus on the most relevant features within each data modality. By incorporating the proposed feature-level attention, we develop a novel Intra-modal multi-head attention (MHA), Inter-modal MHA and Target-modal MHA in a way that both feature and temporal attentions are incorporated in MHAs. This enables the MHAs to generate temporal attentions with consideration of modality and feature importance which leads to more informative embeddings. The proposed modality-aware structure enables the model to effectively exploit information within each modality as well as foster cross-modal understanding. Our extensive experiments on financial datasets demonstrate that Modality-aware Transformer outperforms existing methods, offering a novel and practical solution to the complex challenges of multi-modal financial time series forecasting.