mlsanityData to 13 Oct 2026

GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Values

arXiv 2311.03426 ↗First posted 6 Nov 2023cs.LG cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Massive transformer-based models face several challenges, including slow and computationally intensive pre-training and over-parametrization. This paper addresses these challenges by proposing a versatile method called GQKVA, which generalizes query, key, and value grouping techniques. GQKVA is designed to speed up transformer pre-training while reducing the model size. Our experiments with various GQKVA variants highlight a clear trade-off between performance and model size, allowing for customized choices based on resource and time limitations. Our findings also indicate that the conventional multi-head attention approach is not always the best choice, as there are lighter and faster alternatives available. We tested our method on ViT, which achieved an approximate 0.3% increase in accuracy while reducing the model size by about 4% in the task of image classification. Additionally, our most aggressive model reduction experiment resulted in a reduction of approximately 15% in model size, with only around a 1% drop in accuracy.

Key facts. Cited by 9 papers in the map (data through 13 Oct 2026). Builds on 14 of its 19 references that are in the map. First posted on arXiv 6 Nov 2023.

Builds on

14 of 19 references.

References · 14

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,574
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
  6. 2017Decoupled Weight Decay Regularizationcited by 1,788
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,625
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,469
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,323
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 705
  11. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  12. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
2 more
  1. 2022Token Dropping for Efficient BERT Pretrainingcited by 42
  2. 2023Speeding up Resnet Architecture with Layers Targeted Low Rank Decompositioncited by 8

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 7; 2025: 1; 2026: 1.

9 papers.

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 126
  2. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  3. 2024QCQA: Quality and Capacity-aware grouped Query Attentioncited by 6
  4. 2024SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connectioncited by 5
  5. 2025ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Trainingcited by 3
  6. 2024Beyond Uniform Query Distribution: Key-Driven Grouped Query Attentioncited by 2
  7. 2024Accelerating the Low-Rank Decomposed Modelscited by 1
  8. 2024Is 3D Convolution with 5D Tensors Really Necessary for Video Analysis?cited by 0
  9. 2026EPAS: Efficient Training with Progressive Activation Sharingcited by 0