mlsanityData to 7 Oct 2026

HyDRA: Hierarchical and Dynamic Rank Adaptation for Mobile Vision Language Model

arXiv 2512.20674 ↗First posted 20 Dec 2025cs.LG cs.AI cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision Language Models (VLMs) have undergone significant advancements, particularly with the emergence of mobile-oriented VLMs, which offer a wide range of application scenarios. However, the substantial computational requirements for training these models present a significant obstacle to their practical application. To address this issue, Low-Rank Adaptation (LoRA) has been proposed. Nevertheless, the standard LoRA with a fixed rank lacks sufficient capability for training mobile VLMs that process both text and image modalities. In this work, we introduce HyDRA, a parameter-efficient fine-tuning framework designed to implement hierarchical and dynamic rank scheduling for mobile VLMs. This framework incorporates two essential optimization strategies: (1) hierarchical optimization, which involves a coarse-grained approach that assigns different ranks to various layers, as well as a fine-grained method that adjusts ranks within individual layers, and (2) dynamic adjustment, which employs an end-to-end automatic optimization using a lightweight performance model to determine and adjust ranks during the fine-tuning process. Comprehensive experiments conducted on popular benchmarks demonstrate that HyDRA consistently outperforms the baseline, achieving a 4.7% improvement across various model sizes without increasing the number of trainable parameters. In some tasks, it even surpasses full-parameter fine-tuning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 38 references that are in the map. First posted on arXiv 20 Dec 2025.

Builds on

32 of 38 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2023Mistral 7Bcited by 3,065
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
20 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  3. 2019Towards VQA Models That Can Readcited by 1,696
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  7. 2023A Survey on Multimodal Large Language Modelscited by 898
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  9. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  10. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  11. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  12. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  13. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  14. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  15. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  16. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  17. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  18. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  19. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  20. 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025LangVision-LoRA-NAS: Neural Architecture Search for Variable LoRA Rank in Vision Language Models
  2. 2024Low-Rank Few-Shot Adaptation of Vision-Language Models
  3. 2025LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Models
  4. 2026Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation
  5. 2025Efficient Few-Shot Continual Learning in Vision-Language Models
  6. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Model
  7. 2025Integrating Frequency-Domain Representations with Low-Rank Adaptation in Vision-Language Models
  8. 2023LOVM: Language-Only Vision Model Selection