mlsanityData to 14 Oct 2026

QKV Projections Require a Fraction of Their Memory

arXiv 2506.02939 ↗First posted 3 Jun 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

The Multi-Head Attention mechanism is central to LLM operation, and multiple works target its compute and memory efficiency during training. While most works focus on approximating the scaled dot product, the memory consumption of the linear projections that compute the Q, K, and V tensors from the input x is often overlooked. To address this, we propose Point-Approximate Matrix Multiplication (PAMM), a novel tensor compression technique that compresses the activations of the Q,K,V projections in attention layers by a factor of up to × 512, effectively erasing their memory footprint, while achieving similar or better final perplexity. PAMM is fully composable with efficient attention techniques such as FlashAttention, making it a practical and complementary method for memory-efficient LLM training.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 40 of its 48 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

40 of 48 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  12. 2020Reformer: The Efficient Transformercited by 1,668
28 more
  1. 2020Rethinking Attention with Performerscited by 1,601
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  3. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  4. 2017Mixed Precision Trainingcited by 1,156
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  7. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  8. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  9. 2016AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classificationcited by 347
  10. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  11. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 162
  12. 2023The emergence of clusters in self-attention dynamicscited by 140
  13. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  14. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 45
  15. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 38
  16. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 33
  17. 2022GACT: Activation Compressed Training for Generic Network Architecturescited by 31
  18. 2024Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradientscited by 18
  19. 2021Mesa: A Memory-saving Training Framework for Transformerscited by 16
  20. 2024ESPACE: Dimensionality Reduction of Activations for Model Compressioncited by 15
  21. 2024CompAct: Compressed Activations for Memory-Efficient LLM Trainingcited by 15
  22. 2018Faster Neural Network Training with Approximate Tensor Operationscited by 14
  23. 2022Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformerscited by 11
  24. 2023Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Modelcited by 9
  25. 2024SinkLoRA: Enhanced Efficiency and Chat Capabilities for Long-Context Large Language Modelscited by 9
  26. 2024Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attentioncited by 6
  27. 2024Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagationcited by 5
  28. 2025Exploring Narrative Clustering in Large Language Models: A Layerwise Analysis of BERTcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Low-Rank Key Value Attentioncited by 2
  2. 2026DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisionscited by 1