mlsanityData to 7 Oct 2026

MARS: Harmonizing Multimodal Convergence via Adaptive Rank Search

arXiv 2603.00720 ↗First posted 28 Feb 2026cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Fine-tuning Multimodal Large Language Models (MLLMs) with parameter-efficient methods like Low-Rank Adaptation (LoRA) is crucial for task adaptation. However, imbalanced training dynamics across modalities often lead to suboptimal accuracy due to negative interference, a challenge typically addressed with inefficient heuristic methods such as manually tuning separate learning rates. To overcome this, we introduce MARS (Multimodal Adaptive Rank Search), an approach to discover optimal rank pairs that balance training dynamics while maximizing performance. Our key innovation, a proposed framework of dual scaling laws, enables this search: one law models module-specific convergence time to prune the search space to candidates with aligned dynamics, while the other predicts final task performance to select the optimal pair from the pruned set. By re-purposing the LoRA rank as a controller for modality-specific convergence speed, MARS outperforms baseline methods and provides a robust, automated strategy for optimizing MLLM fine-tuning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 38 references that are in the map. First posted on arXiv 28 Feb 2026.

Builds on

31 of 38 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning30 Sep
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  3. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 26

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  8. 2024Qwen2 Technical Reportcited by 2,076
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  12. 2016A Diagram Is Worth A Dozen Imagescited by 793
14 more
  1. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  2. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  3. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  4. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  5. 2024LoRA Learns Less and Forgets Lesscited by 313
  6. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  7. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  8. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 137
  9. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  10. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 102
  11. 2025Scaling Laws for Native Multimodal Modelscited by 47
  12. 2025On the Convergence Rate of LoRA Gradient Descentcited by 9
  13. 2024GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuningcited by 7
  14. 2021Data-free mixed-precision quantization using novel sensitivity metriccited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
  2. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
  3. 2026Flexi-LoRA with Input-Adaptive Ranks: Efficient Finetuning for Speech and Reasoning Tasks
  4. 2025Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning
  5. 2024CoMMIT: Coordinated Multimodal Instruction Tuning
  6. 2025GoRA: Gradient-driven Adaptive Low Rank Adaptation
  7. 2025LoRA Is Slower Than You Think
  8. 2025MokA: Multimodal Low-Rank Adaptation for MLLMs