mlsanityData to 14 Oct 2026

Transformer-VQ: Linear-Time Transformers via Vector Quantization

arXiv 2309.16354 ↗First posted 28 Sep 2023cs.LG cs.CL cs.CVcited by 26

Not on the map. Why?

Abstract (arXiv)

We introduce Transformer-VQ, a decoder-only transformer computing softmax-based dense self-attention in linear time. Transformer-VQ's efficient attention is enabled by vector-quantized keys and a novel caching mechanism. In our large-scale experiments, Transformer-VQ is shown highly competitive in quality, obtaining 0.99 bpb on Enwik8, 26.6 ppl on PG-19, and 3.16 bpb on ImageNet64. In addition, the optimized implementation of Transformer-VQ is over 3x faster than a comparable quadratic-time transformer at sequence length 8k, is over 12x faster at 32k, and can scale to 131k with similar throughput. Code available: urlhttps://github.com/transformer-vq/transformervq

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 85 of its 104 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

85 of 104 references.

References · 85

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2022Flow Matching for Generative Modelingcited by 5,702
  4. 2016Layer Normalizationcited by 5,669
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
38 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2020Reformer: The Efficient Transformercited by 1,668
  5. 2020Rethinking Attention with Performerscited by 1,601
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  7. 2019Root Mean Square Layer Normalizationcited by 1,480
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  9. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  10. 2021Variational Diffusion Modelscited by 1,153
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  12. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  13. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  14. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  15. 2016Pixel Recurrent Neural Networkscited by 816
  16. 2020Efficient Transformers: A Surveycited by 790
  17. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  18. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  19. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  20. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  21. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  22. 2020Jukebox: A Generative Model for Musiccited by 542
  23. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  24. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  25. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
  26. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  27. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  28. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  29. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
  30. 2021Random Feature Attentioncited by 354
  31. 2021FNet: Mixing Tokens with Fourier Transformscited by 347
  32. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  33. 2020Query-Key Normalization for Transformerscited by 280
  34. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  35. 2022Long Range Language Modeling via Gated State Spacescited by 265
  36. 2020Sparse Sinkhorn Attentioncited by 251
  37. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  38. 2021Hash Layers For Large Sparse Modelscited by 237

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 7; 2026: 7.

26 papers.

Sort:

By area:

  1. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  2. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  3. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  4. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  5. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 31
  6. 2024Scalable Autoregressive Image Generation with Mambacited by 26
  7. 2024Residual vector quantization for KV cache compression in large language modelcited by 7
  8. 2025A2ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantizationcited by 7
  9. 2026On State Reduction in Linear Attentioncited by 6
  10. 2024Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequencescited by 5
  11. 2024LongVQ: Long Sequence Modeling with Vector Quantization on Structured Memorycited by 4
  12. 2025Parallel Sequence Modeling via Generalized Spatial Propagation Networkcited by 4
14 more
  1. 2025Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compressioncited by 4
  2. 2025PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantizationcited by 3
  3. 2025Multi-Sense Embeddings for Language Models and Knowledge Distillationcited by 3
  4. 2025AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Modelscited by 3
  5. 2024Scaling Laws For Mixed Quantizationcited by 2
  6. 2026RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable Statecited by 2
  7. 2024Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddingscited by 1
  8. 2024Context-Aware Token Selection and Packing for Enhanced Vision Transformercited by 1
  9. 2026Online Vector Quantized Attentioncited by 1
  10. 2026Interpretable Vision Transformers in Image Classification via SVDAcited by 1
  11. 2026AVQ-Attention: Adaptive Vector-Quantized Attentioncited by 1
  12. 2025Inter2Former: Dynamic Hybrid Attention for Efficient High-Precision Interactivecited by 0
  13. 2026Enhancing Layer Interaction Using Key-Correlated Layer Attentioncited by 0
  14. 2026PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity
  2. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer
  3. 2026EFQ-Softmax: Exp-Free Quantization for Softmax
  4. 2025Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer
  5. 2025Softpick: No Attention Sink, No Massive Activations with Rectified Softmax
  6. 2026BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers
  7. 2024Trio-ViT: Post-Training Quantization and Acceleration for Softmax-Free Efficient Vision Transformer
  8. 2023Finite Scalar Quantization: VQ-VAE Made Simple