mlsanityData to 7 Oct 2026

Vertical LoRA: Dense Expectation-Maximization Interpretation of Transformers

arXiv 2406.09315 ↗First posted 13 Jun 2024cs.AI cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we show how Transformers can be interpreted as dense Expectation-Maximization algorithms performed on Bayesian Nets. Based on the above interpretation, we propose a new model design paradigm, namely Vertical LoRA (VLoRA), which reduces the parameter count dramatically while preserving performance. In VLoRA, a model consists of layers, each of which recursively learns an increment based on the previous layer. We then apply LoRA decomposition to the increments. VLoRA works on the base model, which is orthogonal to LoRA, meaning they can be used together. We do experiments on various tasks and models. The results show that 1) with VLoRA, the Transformer model parameter count can be reduced dramatically and 2) the performance of the original model is preserved. The source code is available at urlhttps://github.com/neverUseThisName/vlora

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 31 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

26 of 31 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
14 more
  1. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  2. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  3. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  4. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  5. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  6. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  7. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  8. 2013Predicting Parameters in Deep Learningcited by 223
  9. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  10. 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 161
  11. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
  12. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  13. 2024PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimizationcited by 19
  14. 2024LORS: Low-rank Residual Structure for Parameter-Efficient Network Stackingcited by 5

Built on by

Not cited yet.