mlsanityData to 7 Oct 2026

Dataset-Driven Channel Masks in Transformers for Multivariate Time Series

arXiv 2410.23222 ↗First posted 30 Oct 2024cs.LG cs.AI stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

Recent advancements in foundation models have been successfully extended to the time series (TS) domain, facilitated by the emergence of large-scale TS datasets. However, previous efforts have primarily Capturing channel dependency (CD) is essential for modeling multivariate time series (TS), and attention-based methods have been widely employed for this purpose. Nonetheless, these methods primarily focus on modifying the architecture, often neglecting the importance of dataset-specific characteristics. In this work, we introduce the concept of partial channel dependence (PCD) to enhance CD modeling in Transformer-based models by leveraging dataset-specific information to refine the CD captured by the model. To achieve PCD, we propose channel masks (CMs), which are integrated into the attention matrices of Transformers via element-wise multiplication. CMs consist of two components: 1) a similarity matrix that captures relationships between the channels, and 2) dataset-specific and learnable domain parameters that refine the similarity matrix. We validate the effectiveness of PCD across diverse tasks and datasets with various backbones. Code is available at this repository: https://github.com/YonseiML/pcd.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 38 of its 54 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

38 of 54 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  3. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  4. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  5. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
  6. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,030
  7. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 906
  8. 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 869
  9. 2019Graph WaveNet for Deep Spatial-Temporal Graph Modelingcited by 820
  10. 2023A decoder-only foundation model for time-series forecastingcited by 650
  11. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 615
  12. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
26 more
  1. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 453
  2. 2018The UCR Time Series Archivecited by 421
  3. 2018Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholdingcited by 407
  4. 2021SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interactioncited by 335
  5. 2018The UEA multivariate time series classification archive, 2018cited by 330
  6. 2023TSMixer: An All-MLP Architecture for Time Series Forecastingcited by 232
  7. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 198
  8. 2024Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Seriescited by 120
  9. 2024TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysiscited by 113
  10. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  11. 2023Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecastingcited by 102
  12. 2023The Capacity and Robustness Trade-off: Revisiting the Channel Independent Strategy for Multivariate Time Series Forecastingcited by 100
  13. 2023Bake off redux: a review and experimental evaluation of recent time series classification algorithmscited by 99
  14. 2024UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Predictioncited by 86
  15. 2024TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecastingcited by 63
  16. 2023CARD: Channel Aligned Robust Blend Transformer for Time Series Forecastingcited by 55
  17. 2024SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attentioncited by 43
  18. 2024TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysiscited by 37
  19. 2024From Similarity to Superiority: Channel Clustering for Time Series Forecastingcited by 37
  20. 2023Learning to Embed Time Series Patches Independentlycited by 34
  21. 2024Rethinking Channel Dependence for Multivariate Time Series Forecasting: Learning from Leading Indicatorscited by 32
  22. 2024TimeSiam: A Pre-Training Framework for Siamese Time-Series Modelingcited by 20
  23. 2023Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mappingcited by 11
  24. 2024Minusformer: Improving Time Series Forecasting by Progressively Learning Residualscited by 10
  25. 2024PRformer: Pyramidal Recurrent Transformer for Multivariate Time Series Forecastingcited by 5
  26. 2024Enhancing Multivariate Time Series Forecasting with Mutual Information-driven Cross-Variable and Temporal Modelingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026EXAONE Finance 1.0: An Attention-free Time Series Foundation Model for Financial Time Seriescited by 2
  2. 2025LeForecast: Enterprise Hybrid Forecast by Time Series Intelligencecited by 0
  3. 2026EXAONE Demand 1.0: A Time Series Foundation Model for Demand Forecastingcited by 0