# Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling

arXiv: https://arxiv.org/abs/2610.09090 · HTML: https://mlsanity.com/paper/2610.09090/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 68 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Builds on (most cited first, 12 of 12)

- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2412.07772/) (2024)
- [Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models](https://mlsanity.com/paper/2503.09573/) (2025)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [The Origins of Representation Manifolds in Large Language Models](https://mlsanity.com/paper/2505.18235/) (2025)
- [Scalable-Softmax Is Superior for Attention](https://mlsanity.com/paper/2501.19399/) (2025)
- [Tucker Attention: A generalization of approximate attention mechanisms](https://mlsanity.com/paper/2603.30033/) (2026)
- [Higher-Order Transformers With Kronecker-Structured Attention](https://mlsanity.com/paper/2412.02919/) (2024)
- [BlockGPT: Spatio-Temporal Modelling of Rainfall via Frame-Level Autoregression](https://mlsanity.com/paper/2510.06293/) (2025)

## Abstract (arXiv)

The quadratic cost of self-attention limits scalability to long sequences from multidimensional data. We introduce Tucker bottleneck attention (TuBA), which exploits low-rank tensor structure for efficient global token mixing. TuBA projects hidden tensors into compact Tucker cores, performs multi-head self-attention and linear projections on the cores, and writes updates back to the ambient space, enabling subquadratic computation. Its autoregressive extension combines bidirectional interactions within cores with causal attention across cores. On video prediction and global weather forecasting, TuBA achieves favorable accuracy-efficiency trade-offs over standard and efficient attention and task-specific models. Compared to standard self-attention, TuBA reduces error and computation by up to 24.7% and 66.6% for video prediction and 37.1% and 85.1% for autoregressive weather forecasting, with speedups up to 4.27 times. Low-rank Tucker cores and multi-frame generation also outperform full-rank attention and frame-by-frame generation, respectively.