mlsanityData to 7 Oct 2026

GiVA: Gradient-Informed Bases for Vector-Based Adaptation

arXiv 2604.21901 ↗First posted 23 Apr 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

As model sizes continue to grow, parameter-efficient fine-tuning has emerged as a powerful alternative to full fine-tuning. While LoRA is widely adopted among these methods, recent research has explored vector-based adaptation methods due to their extreme parameter efficiency. However, these methods typically require substantially higher ranks than LoRA to match its performance, leading to increased training costs. This work introduces GiVA, a gradient-based initialization strategy for vector-based adaptation. It achieves training times comparable to LoRA and maintains the extreme parameter efficiency of vector-based adaptation. We evaluate GiVA across diverse benchmarks, including natural language understanding, natural language generation, and image classification. Experiments show that our approach consistently outperforms or achieves performance competitive with existing vector-based adaptation methods and LoRA while reducing rank requirements by a factor of eight (8×).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 50 of its 71 references that are in the map. First posted on arXiv 23 Apr 2026.

Builds on

50 of 71 references.

References · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  8. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  9. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  10. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  11. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  12. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  13. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  14. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  15. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  16. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  17. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  18. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  19. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  20. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  21. 2024Mixture of LoRA Expertscited by 179
  22. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  23. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  24. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  25. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 103
  26. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 73
  27. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 70
  28. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 51
  29. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
  30. 2024OLoRA: Orthonormal Low-Rank Adaptation of Large Language Modelscited by 39
  31. 2025LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficientlycited by 35
  32. 2025RandLoRA: Full-rank parameter-efficient fine-tuning of large modelscited by 34
  33. 2023NOLA: Compressing LoRA using Linear Combination of Random Basiscited by 28
  34. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuningcited by 18
  35. 2024Parameter Efficient Fine-tuning via Explained Variance Adaptationcited by 17
  36. 2025GoRA: Gradient-driven Adaptive Low Rank Adaptationcited by 11
  37. 2024GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuningcited by 7
  38. 2025OSoRA: Output-Dimension and Singular-Value Initialized Low-Rank Adaptationcited by 1

Built on by

Not cited yet.