mlsanityData to 7 Oct 2026

ReLope: KL-Regularized LoRA Probes for Multimodal LLM Routing

arXiv 2603.24787 ↗First posted 25 Mar 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Routing balances performance and cost in hybrid systems by escalating selected queries from a lightweight model to a powerful but expensive one. Hidden-state probes provide an effective routing signal by predicting the small model's correctness from hidden states it already computes. Although effective in text-only settings, their behavior on multimodal LLMs (MLLMs) is less understood. We find that correctness is harder to predict from hidden states when questions are paired with images rather than with captions of those images. Two training-free dependence measures, HSIC and label CKA, show the same pattern. We first investigate whether changing hidden states used by the probe improves correctness prediction for MLLMs. We propose Attention Probe, which learns an attention query to pool frozen token states into a routing feature and improves over the standard probe. However, pooling only recombines hidden states computed for answer generation, not for deciding whether the answer should be trusted. We therefore build a decision layer inside the existing MLLM and train it for the routing decision. The resulting router, ReLope (KL-Regularized LoRA Probe), adapts a separate copy of one MLLM layer with LoRA and regularizes it with a KL-penalized stochastic bottleneck. This decision branch shares the frozen lower layers and leaves the model's answer-generation path unchanged. ReLope attains the highest correctness prediction AUC on all five benchmarks and three backbones we evaluate and improves the accuracy versus cost trade-off at negligible overhead. These results suggest a new perspective and a practical technical path on routing: rather than reading decisions off states built for generation, routing decisions can be learned by a dedicated decision layer inside the model. Code: https://github.com/Spinozaaa/ReLope

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 52 references that are in the map. First posted on arXiv 25 Mar 2026.

Builds on

24 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing25 Jan
  2. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs3 Mar

Other references · 22

  1. 2025Qwen2.5-VL Technical Reportcited by 5,418
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  3. 2025Gemma 3 Technical Reportcited by 1,686
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  5. 2016Deep Variational Information Bottleneckcited by 915
  6. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  7. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  8. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  9. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  10. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
  11. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  12. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
10 more
  1. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  2. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  3. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 63
  4. 2024Factual Confidence of LLMs: on Reliability and Robustness of Current Estimatorscited by 59
  5. 2023Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Modelscited by 40
  6. 2024Routoo: Learning to Route to Large Language Models Effectivelycited by 34
  7. 2024Eagle: Efficient Training-Free Router for Multi-LLM Inferencecited by 34
  8. 2025Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalizationcited by 29
  9. 2025Adaptive LLM Routing under Budget Constraintscited by 21
  10. 2025Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Modelscited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers
  2. 2026LoRA-Drop: Temporal LoRA Decoding for Efficient LLM Inference
  3. 2026Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inference
  4. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning
  5. 2025CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization
  6. 2025Multilingual Routing in Mixture-of-Experts
  7. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning
  8. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey