# Transformer-VQ: Linear-Time Transformers via Vector Quantization

arXiv: https://arxiv.org/abs/2309.16354 · HTML: https://mlsanity.com/paper/2309.16354/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 85 of its 104 references that are in the map.
- First posted on arXiv 28 Sep 2023.

## Builds on (most cited first, 30 of 85)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)

## Built on by (most cited first, 26 of 26)

- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [PQCache: Product Quantization-based KVCache for Long Context LLM Inference](https://mlsanity.com/paper/2407.12820/) (2024)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms](https://mlsanity.com/paper/2409.16694/) (2024)
- [Scalable Autoregressive Image Generation with Mamba](https://mlsanity.com/paper/2408.12245/) (2024)
- [Residual vector quantization for KV cache compression in large language model](https://mlsanity.com/paper/2410.15704/) (2024)
- [A2ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantization](https://mlsanity.com/paper/2502.12665/) (2025)
- [On State Reduction in Linear Attention](https://mlsanity.com/paper/2602.04852/) (2026)
- [Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences](https://mlsanity.com/paper/2406.08128/) (2024)
- [LongVQ: Long Sequence Modeling with Vector Quantization on Structured Memory](https://mlsanity.com/paper/2404.11163/) (2024)
- [Parallel Sequence Modeling via Generalized Spatial Propagation Network](https://mlsanity.com/paper/2501.12381/) (2025)
- [Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression](https://mlsanity.com/paper/2510.20984/) (2025)
- [PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization](https://mlsanity.com/paper/2503.06482/) (2025)
- [Multi-Sense Embeddings for Language Models and Knowledge Distillation](https://mlsanity.com/paper/2504.06036/) (2025)
- [AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Models](https://mlsanity.com/paper/2506.19505/) (2025)
- [Scaling Laws For Mixed Quantization](https://mlsanity.com/paper/2410.06722/) (2024)
- [RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State](https://mlsanity.com/paper/2602.11958/) (2026)
- [Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings](https://mlsanity.com/paper/2402.06492/) (2024)
- [Context-Aware Token Selection and Packing for Enhanced Vision Transformer](https://mlsanity.com/paper/2410.23608/) (2024)
- [Online Vector Quantized Attention](https://mlsanity.com/paper/2602.03922/) (2026)
- [Interpretable Vision Transformers in Image Classification via SVDA](https://mlsanity.com/paper/2602.10994/) (2026)
- [AVQ-Attention: Adaptive Vector-Quantized Attention](https://mlsanity.com/paper/2607.12789/) (2026)
- [Inter2Former: Dynamic Hybrid Attention for Efficient High-Precision Interactive](https://mlsanity.com/paper/2507.09612/) (2025)
- [Enhancing Layer Interaction Using Key-Correlated Layer Attention](https://mlsanity.com/paper/2606.28405/) (2026)
- [PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attention](https://mlsanity.com/paper/2609.33746/) (2026)

## Abstract (arXiv)

We introduce Transformer-VQ, a decoder-only transformer computing softmax-based dense self-attention in linear time. Transformer-VQ's efficient attention is enabled by vector-quantized keys and a novel caching mechanism. In our large-scale experiments, Transformer-VQ is shown highly competitive in quality, obtaining 0.99 bpb on Enwik8, 26.6 ppl on PG-19, and 3.16 bpb on ImageNet64. In addition, the optimized implementation of Transformer-VQ is over 3x faster than a comparable quadratic-time transformer at sequence length 8k, is over 12x faster at 32k, and can scale to 131k with similar throughput. Code available: urlhttps://github.com/transformer-vq/transformervq