mlsanityData to 7 Oct 2026

UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Models

arXiv 2505.20154 ↗First posted 26 May 2025cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

This paper introduces Uniform Orthogonal Reinitialization Adaptation (UORA), a novel parameter-efficient fine-tuning (PEFT) approach for Large Language Models (LLMs). UORA achieves state-of-the-art performance and parameter efficiency by leveraging a low-rank approximation method to reduce the number of trainable parameters. Unlike existing methods such as LoRA and VeRA, UORA employs an interpolation-based reparametrization mechanism that selectively reinitializes rows and columns in frozen projection matrices, guided by the vector magnitude heuristic. This results in substantially fewer trainable parameters compared to LoRA and outperforms VeRA in computation and storage efficiency. Comprehensive experiments across various benchmarks demonstrate UORA's superiority in achieving competitive fine-tuning performance with negligible computational overhead. We demonstrate its performance on GLUE and E2E benchmarks and its effectiveness in instruction-tuning large language models and image classification models. Our contributions establish a new paradigm for scalable and resource-efficient fine-tuning of LLMs.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 64 of its 90 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

64 of 90 references.

References · 64

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
38 more
  1. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  2. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  3. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  4. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  5. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  6. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  7. 2017Learning multiple visual domains with residual adapterscited by 553
  8. 2016Solving General Arithmetic Word Problemscited by 353
  9. 2024Graph Retrieval-Augmented Generation: A Surveycited by 320
  10. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  11. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  12. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  13. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  14. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  15. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  16. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  17. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  18. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  19. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  20. 2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
  21. 2025RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?cited by 60
  22. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  23. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steeringcited by 50
  24. 2025CoT-Kinetics: A Theoretical Modeling Assessing LRM Reasoning Processcited by 50
  25. 2024What does the Knowledge Neuron Thesis Have to do with Knowledge?cited by 49
  26. 2019Training Neural Networks for and by Interpolationcited by 45
  27. 2024Disentangling Memory and Reasoning Ability in Large Language Modelscited by 43
  28. 2025Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Modelscited by 43
  29. 2024Advancing Parameter Efficiency in Fine-tuning via Representation Editingcited by 37
  30. 2024Multimodal Instruction Tuning with Conditional Mixture of LoRAcited by 36
  31. 2023Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuningcited by 29
  32. 2023CPSeg: Finer-grained Image Semantic Segmentation via Chain-of-Thought Language Promptingcited by 28
  33. 2024RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructionscited by 25
  34. 2020On the Neural Tangent Kernel of Deep Networks with Orthogonal Initializationcited by 21
  35. 2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuningcited by 19
  36. 2024ResLoRA: Identity Residual Mapping in Low-Rank Adaptioncited by 18
  37. 2025SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignmentcited by 17
  38. 2024Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inferencecited by 15

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025Efficient Agents: Building Effective Agents While Reducing Costcited by 32
  2. 2026Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptationcited by 11
  3. 2026RAM: Recover Any 3D Human Motion in-the-Wildcited by 11
  4. 2025Steering When Necessary: Flexible Steering Large Language Models with Backtrackingcited by 7
  5. 2025RegionMarker: A Region-Triggered Semantic Watermarking Framework for Embedding-as-a-Service Copyright Protectioncited by 4
  6. 2026Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contextscited by 4
  7. 2025Language-Specific Layer Matters: Efficient Multilingual Enhancement for Large Vision-Language Modelscited by 3
  8. 2025Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrievalcited by 2
  9. 2025CCF: A Context Compression Framework for Efficient Long-Sequence Language Modelingcited by 1
  10. 2026Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuningcited by 1
  11. 2026Early Intervention for VFM-based Multimodal Medical Image Classificationcited by 0