mlsanityData to 7 Oct 2026

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention

arXiv 2411.02063 ↗First posted 4 Nov 2024cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Improving the effectiveness and efficiency of large language models (LLMs) simultaneously is a critical yet challenging research goal. In this paper, we find that low-rank pre-training, normally considered as efficient methods that will compromise performance, can be scalably effective when reduced parameters are precisely targeted. Specifically, applying the low-dimensional module only to the attention layer -- resolves this issue and enhances both effectiveness and efficiency. We refer to this structure as Low-dimensional Projected Attention (LPA) and provide an explanatory analysis. Through extensive experimentation at parameter scales of 130M, 370M, and scaling up to 3B, we have validated the effectiveness and scalability of LPA. Our results show that LPA model can save up to 12.4% in time while achieving an approximate 5% improvement in test perplexity (ppl) and on downstream tasks compared with the vanilla Transformer.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 27 of its 43 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

27 of 43 references.

References · 27

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2016Layer Normalizationcited by 5,579
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
15 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  5. 2018Neural Network Acceptability Judgmentscited by 952
  6. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  7. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  8. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  9. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  10. 2020Low-Rank Bottleneck in Multi-head Attention Modelscited by 88
  11. 2020Growing Efficient Deep Networks by Structured Continuous Sparsificationcited by 28
  12. 2023SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Modelscited by 26
  13. 2022Sparse Structure Search for Parameter-Efficient Tuningcited by 16
  14. 2021DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Modelscited by 8
  15. 2020HOTCAKE: Higher Order Tucker Articulated Kernels for Deeper CNN Compressioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Slamming: Training a Speech Language Model on One GPU in a Daycited by 11
  2. 2025Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarkingcited by 4
  3. 2025Efficient Low Rank Attention for Long-Context Inference in Large Language Modelscited by 4
  4. 2025QKV Projections Require a Fraction of Their Memorycited by 2
  5. 2026Low-Rank Key Value Attentioncited by 1
  6. 2026ARM: Attention with Routed-Memory for Learnable Sparse Controlcited by 0