# Dataset-Driven Channel Masks in Transformers for Multivariate Time Series

arXiv: https://arxiv.org/abs/2410.23222 · HTML: https://mlsanity.com/paper/2410.23222/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 54 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis](https://mlsanity.com/paper/2210.02186/) (2022)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Graph WaveNet for Deep Spatial-Temporal Graph Modeling](https://mlsanity.com/paper/1906.00121/) (2019)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding](https://mlsanity.com/paper/1802.04431/) (2018)
- [The UEA multivariate time series classification archive, 2018](https://mlsanity.com/paper/1811.00075/) (2018)
- [SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction](https://mlsanity.com/paper/2106.09305/) (2021)
- [TSMixer: An All-MLP Architecture for Time Series Forecasting](https://mlsanity.com/paper/2303.06053/) (2023)
- [Timer: Generative Pre-trained Transformers Are Large Time Series Models](https://mlsanity.com/paper/2402.02368/) (2024)
- [Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series](https://mlsanity.com/paper/2401.03955/) (2024)
- [TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysis](https://mlsanity.com/paper/2410.16032/) (2024)
- [Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2310.08278/) (2023)
- [Long-Short Transformer: Efficient Transformers for Language and Vision](https://mlsanity.com/paper/2107.02192/) (2021)
- [The Capacity and Robustness Trade-off: Revisiting the Channel Independent Strategy for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2304.05206/) (2023)
- [Bake off redux: a review and experimental evaluation of recent time series classification algorithms](https://mlsanity.com/paper/2304.13029/) (2023)
- [UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Prediction](https://mlsanity.com/paper/2402.11838/) (2024)
- [TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecasting](https://mlsanity.com/paper/2403.09898/) (2024)
- [CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting](https://mlsanity.com/paper/2305.12095/) (2023)
- [SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention](https://mlsanity.com/paper/2402.10198/) (2024)
- [TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis](https://mlsanity.com/paper/2402.16412/) (2024)

## Built on by (most cited first, 3 of 3)

- [EXAONE Finance 1.0: An Attention-free Time Series Foundation Model for Financial Time Series](https://mlsanity.com/paper/2609.04239/) (2026)
- [LeForecast: Enterprise Hybrid Forecast by Time Series Intelligence](https://mlsanity.com/paper/2503.22747/) (2025)
- [EXAONE Demand 1.0: A Time Series Foundation Model for Demand Forecasting](https://mlsanity.com/paper/2609.30880/) (2026)

## Abstract (arXiv)

Recent advancements in foundation models have been successfully extended to the time series (TS) domain, facilitated by the emergence of large-scale TS datasets. However, previous efforts have primarily Capturing channel dependency (CD) is essential for modeling multivariate time series (TS), and attention-based methods have been widely employed for this purpose. Nonetheless, these methods primarily focus on modifying the architecture, often neglecting the importance of dataset-specific characteristics. In this work, we introduce the concept of partial channel dependence (PCD) to enhance CD modeling in Transformer-based models by leveraging dataset-specific information to refine the CD captured by the model. To achieve PCD, we propose channel masks (CMs), which are integrated into the attention matrices of Transformers via element-wise multiplication. CMs consist of two components: 1) a similarity matrix that captures relationships between the channels, and 2) dataset-specific and learnable domain parameters that refine the similarity matrix. We validate the effectiveness of PCD across diverse tasks and datasets with various backbones. Code is available at this repository: https://github.com/YonseiML/pcd.