mlsanityData to 14 Oct 2026

Pooling Helps, Learned Weighting Hurts In-Context: Decomposing Group Attention

arXiv 2610.01831 ↗First posted 1 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Group attention, introduced by the time series forecasting model Chronos-2, attends over the variates of a group at a fixed patch index and serves both multivariate (MV) and in-context learning (ICL) forecasting. Rather than evaluating this cross-variate attention design as a whole, we ask which part of the mechanism earns the benefit and probe its applicability to both MV and ICL regimes. By editing the attention matrix α at inference we separate the two pathways a head comprises: V/O, which projects a weighted summary of the group, and Q/K, which decides the weights. Uniform pooling (V/O without any Q/K weighting) is positive on 18 of our 20 sensor-network configurations, while the learned weighting (Q/K) splits by group type: its contribution is positive or negligible for MV, but materially degrades 8 of the 10 sensor-network ICL configurations, leaving 4 of them worse than univariate inference. By isolating the impact of different layers, we find that uniforming α in the first block alone improves every ICL configuration we test.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 22 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

21 of 22 references.

References · 21

  1. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  2. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  3. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  4. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 931
  5. 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 879
  6. 2019Attention is not not Explanationcited by 636
  7. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 614
  8. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  9. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  10. 2025Chronos-2: From Univariate to Universal Forecastingcited by 194
  11. 2023The Capacity and Robustness Trade-off: Revisiting the Channel Independent Strategy for Multivariate Time Series Forecastingcited by 101
  12. 2024SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusioncited by 76
9 more
  1. 2025This Time is Different: An Observability Perspective on Time Series Foundation Modelscited by 68
  2. 2024SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attentioncited by 44
  3. 2020Hard-Coded Gaussian Attention for Neural Machine Translationcited by 42
  4. 2024From Similarity to Superiority: Channel Clustering for Time Series Forecastingcited by 37
  5. 2024Are Self-Attentions Effective for Time Series Forecasting?cited by 19
  6. 2026TiRex-2: Generalizing TiRex to Multivariate Data and Streamingcited by 13
  7. 2026Universal Redundancies in Time Series Foundation Modelscited by 5
  8. 2024AverageTime: Enhance Long-Term Time Series Forecasting with Simple Averagingcited by 1
  9. 2026Do We Really Need Adaptive Global Spatial Attention for Traffic Forecasting?cited by 1

Built on by

Not cited yet.