# Pooling Helps, Learned Weighting Hurts In-Context: Decomposing Group Attention

arXiv: https://arxiv.org/abs/2610.01831 · HTML: https://mlsanity.com/paper/2610.01831/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 22 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Builds on (most cited first, 21 of 21)

- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [Chronos-2: From Univariate to Universal Forecasting](https://mlsanity.com/paper/2510.15821/) (2025)
- [The Capacity and Robustness Trade-off: Revisiting the Channel Independent Strategy for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2304.05206/) (2023)
- [SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusion](https://mlsanity.com/paper/2404.14197/) (2024)
- [This Time is Different: An Observability Perspective on Time Series Foundation Models](https://mlsanity.com/paper/2505.14766/) (2025)
- [SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention](https://mlsanity.com/paper/2402.10198/) (2024)
- [Hard-Coded Gaussian Attention for Neural Machine Translation](https://mlsanity.com/paper/2005.00742/) (2020)
- [From Similarity to Superiority: Channel Clustering for Time Series Forecasting](https://mlsanity.com/paper/2404.01340/) (2024)
- [Are Self-Attentions Effective for Time Series Forecasting?](https://mlsanity.com/paper/2405.16877/) (2024)
- [TiRex-2: Generalizing TiRex to Multivariate Data and Streaming](https://mlsanity.com/paper/2607.01204/) (2026)
- [Universal Redundancies in Time Series Foundation Models](https://mlsanity.com/paper/2602.01605/) (2026)
- [AverageTime: Enhance Long-Term Time Series Forecasting with Simple Averaging](https://mlsanity.com/paper/2412.20727/) (2024)
- [Do We Really Need Adaptive Global Spatial Attention for Traffic Forecasting?](https://mlsanity.com/paper/2607.12462/) (2026)

## Abstract (arXiv)

Group attention, introduced by the time series forecasting model Chronos-2, attends over the variates of a group at a fixed patch index and serves both multivariate (MV) and in-context learning (ICL) forecasting. Rather than evaluating this cross-variate attention design as a whole, we ask which part of the mechanism earns the benefit and probe its applicability to both MV and ICL regimes. By editing the attention matrix α at inference we separate the two pathways a head comprises: V/O, which projects a weighted summary of the group, and Q/K, which decides the weights. Uniform pooling (V/O without any Q/K weighting) is positive on 18 of our 20 sensor-network configurations, while the learned weighting (Q/K) splits by group type: its contribution is positive or negligible for MV, but materially degrades 8 of the 10 sensor-network ICL configurations, leaving 4 of them worse than univariate inference. By isolating the impact of different layers, we find that uniforming α in the first block alone improves every ICL configuration we test.