# Tensor Product Attention Is All You Need

arXiv: https://arxiv.org/abs/2501.06425 · HTML: https://mlsanity.com/paper/2501.06425/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 73 references that are in the map.
- First posted on arXiv 11 Jan 2025.

## Builds on (most cited first, 30 of 59)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)

## Built on by (most cited first, 30 of 31)

- [Cartridges: Lightweight and general-purpose long context representations via self-study](https://mlsanity.com/paper/2506.06266/) (2025)
- [TransMLA: Multi-Head Latent Attention Is All You Need](https://mlsanity.com/paper/2502.07864/) (2025)
- [Theoretical Guarantees for High Order Trajectory Refinement in Generative Flows](https://mlsanity.com/paper/2503.09069/) (2025)
- [Hardware-Efficient Attention for Fast Decoding](https://mlsanity.com/paper/2505.21487/) (2025)
- [Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?](https://mlsanity.com/paper/2506.17121/) (2025)
- [UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression](https://mlsanity.com/paper/2509.15763/) (2025)
- [Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference](https://mlsanity.com/paper/2505.22913/) (2025)
- [AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping](https://mlsanity.com/paper/2502.11034/) (2025)
- [Quantum-Enhanced LLM Efficient Fine Tuning](https://mlsanity.com/paper/2503.12790/) (2025)
- [Neural Algorithmic Reasoning for Hypergraphs with Looped Transformers](https://mlsanity.com/paper/2501.10688/) (2025)
- [SAS: Simulated Attention Score](https://mlsanity.com/paper/2507.07694/) (2025)
- [Tucker Attention: A generalization of approximate attention mechanisms](https://mlsanity.com/paper/2603.30033/) (2026)
- [Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decoding](https://mlsanity.com/paper/2503.11108/) (2025)
- [Theoretical Foundation of Flow-Based Time Series Generation: Provable Approximation, Generalization, and Efficiency](https://mlsanity.com/paper/2503.14076/) (2025)
- [Low-Rank Key Value Attention](https://mlsanity.com/paper/2601.11471/) (2026)
- [Explicit Multi-head Attention for Inter-head Interaction in Large Language Models](https://mlsanity.com/paper/2601.19611/) (2026)
- [Multi-Head Low-Rank Attention](https://mlsanity.com/paper/2603.02188/) (2026)
- [Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models](https://mlsanity.com/paper/2501.13428/) (2025)
- [Union of Experts: Adapting Hierarchical Routing to Equivalently Decomposed Transformer](https://mlsanity.com/paper/2503.02495/) (2025)
- [WaveFilter: Enhancing the Long-Context Capability of Diffusion LLMs via Wavelet-Guided KV Cache Filtering](https://mlsanity.com/paper/2606.00724/) (2026)
- [Physical models realizing the transformer architecture of large language models](https://mlsanity.com/paper/2507.13354/) (2025)
- [Exact Causal Attention with 10% Fewer Operations](https://mlsanity.com/paper/2510.05175/) (2025)
- [M2I2HA: Multi-modal Object Detection Based on Intra- and Inter-Modal Hypergraph Attention](https://mlsanity.com/paper/2601.14776/) (2026)
- [Product Interaction: An Algebraic Formalism for Deep Learning Architectures](https://mlsanity.com/paper/2602.02573/) (2026)
- [Cubit: Token Mixer with Kernel Ridge Regression](https://mlsanity.com/paper/2605.06501/) (2026)
- [LESSViT: Robust Hyperspectral Representation Learning under Spectral Configuration Shift](https://mlsanity.com/paper/2605.18541/) (2026)
- [When AI meets quantum information: A comprehensive review](https://mlsanity.com/paper/2607.00365/) (2026)
- [From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers](https://mlsanity.com/paper/2607.10677/) (2026)
- [Optimal Power Allocation and AI Receiver Design for Superimposed DMRS and Data Transmission](https://mlsanity.com/paper/2608.13809/) (2026)
- [Kronecker Product Attention: Memory-Efficient KV Compression via Structured Factorization](https://mlsanity.com/paper/2610.12657/) (2026)

## Abstract (arXiv)

Scaling language models to handle longer input sequences typically necessitates large key-value (KV) caches, resulting in substantial memory overhead during inference. In this paper, we propose Tensor Product Attention (TPA), a novel attention mechanism that uses tensor decompositions to represent queries, keys, and values compactly, substantially shrinking the KV cache size at inference time. By factorizing these representations into contextual low-rank components and seamlessly integrating with Rotary Position Embedding (RoPE), TPA achieves improved model quality alongside memory efficiency. Based on TPA, we introduce the Tensor ProducT ATTenTion Transformer (T6), a new model architecture for sequence modeling. Through extensive empirical evaluation on language modeling tasks, we demonstrate that T6 surpasses or matches the performance of standard Transformer baselines including Multi-Head Attention (MHA), Multi-Query Attention (MQA), Grouped-Query Attention (GQA), and Multi-Head Latent Attention (MLA) across various metrics, including perplexity and a range of established evaluation benchmarks. Notably, TPA's memory efficiency and computational efficiency at decoding stage enables processing longer sequences under fixed resource constraints, addressing a critical scalability challenge in modern language models. Project Page: https://github.com/tensorgi/TPA.