# Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together

arXiv: https://arxiv.org/abs/1805.00912 · HTML: https://mlsanity.com/paper/1805.00912/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2018.

## Built on by (most cited first, 5 of 5)

- [UnifieR: A Unified Retriever for Large-Scale Retrieval](https://mlsanity.com/paper/2205.11194/) (2022)
- [TENT: Tensorized Encoder Transformer for Temperature Forecasting](https://mlsanity.com/paper/2106.14742/) (2021)
- [Integral Transformer: Denoising Attention, Not Too Much Not Too Little](https://mlsanity.com/paper/2508.18387/) (2025)
- [Free Energy Mixer](https://mlsanity.com/paper/2602.07160/) (2026)
- [Time Series Representation Learning with Supervised Contrastive Temporal Transformer](https://mlsanity.com/paper/2403.10787/) (2024)

## Abstract (arXiv)

Neural networks equipped with self-attention have parallelizable computation, light-weight structure, and the ability to capture both long-range and local dependencies. Further, their expressive power and performance can be boosted by using a vector to measure pairwise dependency, but this requires to expand the alignment matrix to a tensor, which results in memory and computation bottlenecks. In this paper, we propose a novel attention mechanism called "Multi-mask Tensorized Self-Attention" (MTSA), which is as fast and as memory-efficient as a CNN, but significantly outperforms previous CNN-/RNN-/attention-based models. MTSA 1) captures both pairwise (token2token) and global (source2token) dependencies by a novel compatibility function composed of dot-product and additive attentions, 2) uses a tensor to represent the feature-wise alignment scores for better expressive power but only requires parallelizable matrix multiplications, and 3) combines multi-head with multi-dimensional attentions, and applies a distinct positional mask to each head (subspace), so the memory and computation can be distributed to multiple heads, each with sequential information encoded independently. The experiments show that a CNN/RNN-free model based on MTSA achieves state-of-the-art or competitive performance on nine NLP benchmarks with compelling memory- and time-efficiency.